It seems to just be standard "normalization of deviance" to use the language of safety engineering. You have 5 layers of fallbacks, so over time skipping any of the middle layers doesn't really have anything fail. So in time you end up with a true safety factor equal only to the last layer. Then that fails and looking back "everything had to go wrong".

As Sidney Dekker (of Understanding Human Error fame) says: Murphy's Law is wrong - everything that can go wrong will go right. The problem arises from the operators all assuming that it will keep going right.

I remember reading somewhere that part of Qantas's safety record came from the fact that at one time they had the highest number of minor issues. In some sense, you want your error detection curve to be smooth: as you get closer to catastrophe, your warnings should get more severe. On this ship, it appeared everything was A-OK till it bonked a bridge.

Oh, it gets even worse!

The NTSB also had some comments on the ship's equivalent of a black box. Turns out it was impossible to download the data while it was still inside the ship, the manufacturer's software was awful and the various agencies had a group chat to share 3rd party software(!), the software exported thousands of separate files, audio tracks were mixed to the point of being nearly unusable, and the black box stopped recording some metrics after power loss "because it wasn't required to" - despite the data still being available.

At least they didn't have anything negative to say about the crew: they reacted timely and adequately - they just didn't stand a chance.

One of the things Sal Mercogliano stressed is that the crew (and possibly other crews of the same line) modified systems in order to save time.

Rather than doing the process of purging high-sulphur fuel that can't be used in USA waters, they had it set so that some of the generators were fed from USA-approved fuel, resulting in redundancy & automatic failover being compromised.

It seems probable that the wire failure would not have caused catastrophic overall loss of power if the generators had been in the normal configuration.

Also the zeroth failure mode: someone built a bridge that will collapse if any of the many many large ships that sail beneath it can't steer itself with high precision.

It wasn't. They were feeding generators 1 & 2 with the pump intended for flushing the lines while switching between different fuel types.

The regular fuel pumps were set up to automatically restart, which is why a set of them came online to feed generator 3 (which automatically spinned up after 1 & 2 failed, and wasn't tied to the fuel-line-flushing pump) after the second blackout.

If I see a RJ45 plug with a broken locking thingie, or bare wires (not just bare copper - any internal wire), I chop the plug off.

If I come across a CATx (solid core) cable being used as a really long patch lead then I lose my shit or perhaps get a backbox and face plate and modules out along with a POST tool.

I don't look after floating fires.

I recently had a home network outage. The last thing I tested was the in-wall wiring because I just didn't think that would be the cause. It was. Wiring fails!

If I had a nickle for every time someone clobbered some critical connectivity with an ill-advised switch configuration I wouldn't have to work for a living.

And the physical layer issues I do see are related to ham fisted people doing unrelated work in the cage.

Actual failures are pretty damn rare.

Why? It's cost them 100M (https://www.justice.gov/archives/opa/pr/us-reaches-settlemen...) but rebuilding the bridge is going to be 5.2Billion so if gundecking all this maintenance for 20+ years has saved more then 100M, they will do it again.

It’s not thought. These situations are extremely rare. When they happen it just close the company and shed liability.