Remix.run Logo
▲ montagg a day ago

They are at least interconnected in the sense that we are struggling societally to do big, hard things that we could do a decade ago.

Roman citizen wondering why the aqueduct isn’t working kind of thing.

▲forshaper a day ago | parent | next [-]

Interestingly enough it took about four generations, a century, for someone to go back in and investigate the aqueducts properly. I think a lot of this is due to previously easy, simple things being ignored, in part because they are considered menial. The basics of maintenance are easily forgotten.

▲noir_lord a day ago | parent | next [-]

> The basics of maintenance are easily forgotten.

Also undervalued by society as a whole and politicians especially.

They'd rather allocate funding for a new bridge they can point at and say "I did that" than properly fund the unsexy work of making sure the existing bridges are maintained.

Now apply that across most western democracies, repeat for decades and we are where we are.

The countries doing their vast infrastructure build outs will likely repeat the pattern in a few decades.

▲camillomiller 18 hours ago | parent | prev [-]

The basics of maintenance are... easily cut by VP Chadson Chadenberg, former McKinsey, in his cocaine-fueled spree to increase profitability, you mean?

▲hn_throwaway_99 a day ago | parent | prev | next [-]

The other issue though is that with 24/7 connectivity, any unplanned downtime can feel like a major outage. 50 years ago there were lots of outages, to, but people didn't care as much, they just went outside.

▲jzb a day ago | parent | next [-]

50 years ago, an outage in Seattle didn't affect me in St. Louis as a rule. I'd go to the bank, hand the teller a check, they'd eyeball it, maybe hand me some cash, and process all the things on paper.

When banks started moving to computerized systems, maybe a bank would be unable to process something... but not multiple banks. I can't recall any "outages" that affected banks nationwide for any period of time. Walk into your local bank and find out they can't do something because their big system at the mothership is down? Perhaps.

Consolidation likely plays a part in this as well. I can't speak to the FAA/ATC systems 50 years ago, I have no recollection of what they were like then, but the only nationwide system I can think of around that era was Ma Bell. And I don't honestly recall any nationwide outages back then. Regional, sure. You might have a problem with part of the country getting calls if there was some kind of emergency... (e.g., everybody trying to call their grandmas in Kansas after reports of a tornado) but I don't remember there being any nation-wide problems like that.

We had to count on actual snow for snowdays. :-)

▲mkjs a day ago | parent | next [-]

The manual processes don't exist anymore. If the IT is down, nothing can happen.

▲toast0 a day ago | parent | prev [-]

> And I don't honestly recall any nationwide outages back then

There was this one in 1990 [1], but it's notable because the system was otherwise very reliable. Also, this report says only 50% of call attempts failed, presumably most people would try their call a few times and likely get through.

[1] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...

▲Taikonerd a day ago | parent | prev | next [-]

I would also say that there are more single points of failure on the internet than there were 20 years ago.

In 2006, Amazon had just launched this new storage service called "S3." It would have struck people in 2006 as bizarre to think that AWS (and Cloudflare, and GCP, and friends) would one day be so crucial to the internet that a bug there could take down dozens or hundreds of companies.

▲annzabelle a day ago | parent [-]

And for a lot of companies (I know this is true for Capital One), the sole redundancy plan for the vast majority of services when US-East-1 fails is to fail everything you can over to a different AWS region.

▲getpokedagain 21 hours ago | parent [-]

I'm not surprised. The same type of plan can cover fedramp certification.

▲annzabelle 21 hours ago | parent [-]

That was a frequent question asked at the annual CEO-on-a-stage-in-front-of-the-whole-company event, whether we could be cloud vendor agnostic and setup for an easy lift and shift to another cloud provider, and every year we'd get an answer about how great AWS vendor-lock was for us.

At this point I'm pretty sure 99% of businesses have looked at whether to be ready for a lift and shift between vendors or whether to cut this month's compute budget by using more provider specific services, and pretty much all of them have gone with going with more provider specific services.

▲jamesjhare a day ago | parent | prev | next [-]

50 years ago was 1976. Basically nobody had a computer at home. "Outages" were not a thing.

▲maccard a day ago | parent | next [-]

50 years ago my parents were getting power outages on the somewhat regular. I have had two in the last decade.

▲olyjohn a day ago | parent [-]

I had one last week. And multiple each winter. Its highly environment specific. Unless you're saying you still live in their house?

▲maccard 16 hours ago | parent | next [-]

Very much region dependent. I grew up in Ireland and currently live in the UK in a city. Power outages are extremely rare in both.

▲zenithirl 14 hours ago | parent [-]

Power outages rare in Ireland? Absolutely not. Unless by rare you mean a few per year.

▲Mezzie a day ago | parent | prev | next [-]

I had one this morning. I live in the middle of a metro area of half a million people.

▲duderific a day ago | parent | prev [-]

PG&E I presume? We get multiple per month at this point

▲hn_throwaway_99 a day ago | parent | prev [-]

We still had a banking system, we still had phone lines, etc. But yes, the general lack of an expectation of always-on connectivity meant that service interruptions were much less of a big deal.

▲expedition32 a day ago | parent | prev | next [-]

Fifty years ago the store owner just wrote down your name and you could pay the bill next week.

▲Mezzie a day ago | parent [-]

One of the coolest things I've seen, system design wise, was working at libraries when the tech/power is out for a given period.

Every service except digital lending/materials continued to function, and there were fallback processes for recording everything that happened so the information could be added back into the digital system once the tech was fixed/power was restored. Specifically designed forms that were easy to use and easy to scan, an exact understanding of what needed to be recorded and why and where it would go later, etc. And a staff of people who were meticulous enough to actually follow through.

▲nunez a day ago | parent [-]

This makes sense given that so much of how information theory and data processing works were influenced by libraries, which have existed for thousands of years

▲jojobas a day ago | parent | prev | next [-]

Anyone who hasn't embraced the 24/7 connectivity has been priced out.

▲jeremyjh a day ago | parent | prev [-]

That isn’t something that has changed in the last 10 years.

▲hn_throwaway_99 a day ago | parent [-]

I originally missed OP's comment about "a decade ago", but I don't think our "ability to get big things done" has changed in the last 10 years, nor the number of major outages.

When I hear people talk about "we can't do big things anymore" in the US, they are usually referring to mid 20th century stuff a la https://patrickcollison.com/fast, not stuff that we did 10 years ago.

▲jeremyjh a day ago | parent [-]

The topic of TFA and this thread is that these major outages appear to be happening more frequently this year - or in the last month - than we've seen in the past.

I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad. So that is possibly an example of a hard thing we could do a decade ago, that we can't do now.

▲hn_throwaway_99 a day ago | parent | next [-]

It may appear, but it's not.

> I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad.

In 2023 the FAA grounded the entire country due to a faulty database update. So yeah, these things happen, people, like you, just forget about them.

▲brewdad 20 hours ago | parent [-]

Since 9/11, there have been two nationwide ground stops that lasted more than an hour. Both have happened within the past three years. If we get a third soon, we might well have a trend.

▲hunter2_ a day ago | parent | prev [-]

Deciding on the ground stop instead of some other type of quick rollback might've been more to do with furthering financial agendas by making a scene (under the guise of "an abundance of caution" which does seem like a more popular thing lately than decades ago) as opposed to it actually being the only reasonable action.

▲jeremyjh a day ago | parent [-]

No, that did not happen because every communication related to this is recorded, and will be audited in an investigation, and this would have been known to everyone ahead of time. Most likely the NSTB will investigate, though this has not been announced. There are also going to be congressional hearings and career civil-servants will be subpoenaed.

▲randomNumber7 a day ago | parent | prev | next [-]

It doesn't seem hard to use two angle of attack sensors if you pitch the nose of a plane down based on its output.

▲1270018080 a day ago | parent | prev | next [-]

The thing done today weren't even computationally possible a decade ago. The bar for competitive software is rising exponentially.

▲pdonis a day ago | parent | next [-]

> The thing done today weren't even computationally possible a decade ago.

Not the things that are breaking, no. The FAA was doing ATC a decade ago. We had email and cloud servers a decade ago. We had bank websites a decade ago. The scale of those things today is perhaps larger than a decade ago, but not enormously larger; indeed, the increase in scale in those things is much smaller than the increase in computing power in the world from a decade ago. So these things should have gotten more reliable--but they haven't.

▲ranger_danger a day ago | parent | next [-]

Perhaps we would be better off if businesses simply couldn't get this big.

▲nick486 a day ago | parent | prev | next [-]

we skill-squeezed ourselves as an industry.

we made these systems simpler to maintain, to reduce the amount of skill we need to pay for to maintain them. And that does reduce the maintenance costs, at the cost of more compute. but who cares about compute costs where compute per unit of money doubles every 18 months, amirite?

unfortunately it also increases the competence required to change these systems without degrading their performance and usability.

since no one wants to pay that extra cost either, we let the systems degrade a little bit at every change instead, hoping they degrade slower than hardware improves.

that bet was bound to lose eventually, because of laws of physics, and because that degradation pattern affects hardware too.

and the double whammy is - with little demand for expertise - the supply is effectively gone too. so now that the walls have closed in, there is no way out.

▲randomNumber7 a day ago | parent | prev [-]

We also had this decentralized network which turned into a monopolized shithole that fries normies brains. Although that was probably more than a decade ago.

▲sdevonoes a day ago | parent | prev | next [-]

Important stuff is the same. Banking doesn’t need fancy stuff

▲alex0015 a day ago | parent [-]

Do you think someone could take a look at the things you do for work and accurately say whether it needs "fancy stuff?"

▲mixdup a day ago | parent | prev | next [-]

Both the FAA and the Microsoft services that failed have been around longer than a decade

▲nimih a day ago | parent | prev | next [-]

I'm pretty sure I was able to do online banking a decade ago. USAA even had working 2FA at that time.

▲jghn a day ago | parent | prev | next [-]

> The thing done today weren't even computationally possible a decade ago

I was able to log into my bank's website in the late 90s, FWIW. Yet today I can't seem to do that.

▲Avshalom a day ago | parent | prev [-]

No we definitely had banking software in 2016.

▲ragequittah a day ago | parent | next [-]

We definitely did. And stuff like this happened back then too [1]. And even worse stuff before then [2].

[1]https://www.fintechfutures.com/baas/bt-fault-hits-uk-banks-d...

[2]https://abcnews.com/US/hackers-block-us-banks-online-service...

▲fasterik a day ago | parent | prev | next [-]

"Nothing existed before I was an adult" type vibes.

▲whateveracct a day ago | parent | prev [-]

AI revisionism is out of control generally tbh lol

▲hunterpayne a day ago | parent [-]

I've been working in software for 30 years. This is the largest collection of systems to be down at the same time in that period. I have been literally experiencing a degradation in online services over the last year or two. Its either poor quality AI generated code, or your generation is just incompetent. Your call...

If I were you (or was working at somewhere with an outage), even if it wasn't AI, I would still be blaming AI. Because the alternative means you shouldn't have a job in tech.

▲Telaneo 11 hours ago | parent | prev | next [-]

Where's Agrippa when you need him as aedile?

▲convolvatron a day ago | parent | prev | next [-]

that's not really a good characterization. we have known this was a house of cards for decades. someone just turned on the fan.

▲DIE_DIE_DIE a day ago | parent [-]

[dead]

▲ a day ago | parent | prev | next [-]
[deleted]
▲bongofizzbuzz a day ago | parent | prev [-]

[flagged]

▲rogerrogerr a day ago | parent [-]

What do people mean when they say this?

▲GenerocUsername a day ago | parent | next [-]

They mean there is a large multi-decade conspiracy theory which ties many world events into one cohesive narrative which culminates in a large scale planned destruction of western power structures wherein that controlled demolition scares the proletariat into being accepting of a new world order in which the global currencies are consolidated to a singular digital currency. This shift basically marks the end of individual freedom since nothing short of a uncontrolled global collapse could ever unseat the hedgimony it would create.

▲NoidFonsense a day ago | parent [-]

How could the CBDCs possibly be worse than EMV already is? The world of social credit scores and political debanking exists, unless you are buying coke off the darknet with monero.

But for everything else you need, food water shelter healthcare, you need to submit to europay visa mastercard

▲GenerocUsername a day ago | parent [-]

Yeah, it mostly exists now... And gets closer to the end state every year.

▲monegator a day ago | parent | prev | next [-]

ending of fight club?

▲mathfailure a day ago | parent | prev [-]

[flagged]