Remix.run Logo
▲ bigDinosaur 21 hours ago

I can't think of any other engineering discipline where demand might increase by a factor of a million in a few seconds because some DNS record was propagated incorrectly or something - regulation won't magically solve the issues unique to software. Maybe it's doable with vastly increased costs (a classic would be running fully redundant 1:1 warm failovers, which nobody could economically do at the cloud provider level.)

That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.

▲tarsierscripter 20 hours ago | parent | next [-]

Electrical engineering of the power grid has the exact same failure mode: cascading failures of interconnected power grids can cause loads to increase by factor of a million in a few seconds.

I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.

▲cowlevel 13 hours ago | parent [-]

Also known as *the* blackout. https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, which would have provided ample time to intervene. However, a race condition caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.

Also note the power companies have clear procedures to reboot the grid, known as a "black start[up]". And they test these procedures, though without taking down the grid.

▲jimnotgym 15 hours ago | parent | prev [-]

Which is the kind of exceptionalism that really creates these problems. Software is the only industry that allows people untrained in running global infrastructure to run global infrastructure.

▲NetMageSCW 6 hours ago | parent [-]

So why do other global systems have massive failures?