| ▲ | JsonDemWitOster 3 hours ago |
| I know the leg I'm standing on is correlation but did we have such widespread outages pre-LLMs? The two that come to mind is that one time some intern pushed a config change that borked us-east-1 and Gitlab deleting their DB. And both had clear comms on the situation within acceptable time frames. Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason. |
|
| ▲ | dghlsakjg 3 hours ago | parent | next [-] |
| Large outages are not a new thing. People were making memes about the GitHub unicorn a decade ago. |
| |
| ▲ | JohnMakin 3 hours ago | parent [-] | | No one is saying they are, they’re asking whether the frequency and scale has changed. | | |
| ▲ | tadfisher 3 hours ago | parent | next [-] | | With GitHub specifically, yes. But I remember AWS/GCP/Azure outages were like a yearly occurrence before the 2020s, and we kept having the same conversations back then about infrastructure resilience, multi-cloud, etc. that were too hard to address then and still too expensive to address now. | | |
| ▲ | jeremyjh 3 hours ago | parent [-] | | Maybe one major platform had a major outage annually but AWS certainly didn't have a major outage every year. More like every 3-4 years. I remember exactly two in the last 10 years that impacted the production application I run. | | |
| ▲ | dghlsakjg 19 minutes ago | parent | next [-] | | Here’s a timeline of major AWS outages: https://awsmaniac.com/aws-outages/ Just because an outage doesn’t affect your specific app doesn’t mean it didn’t happen. By that logic Microsoft didn’t have an outage last week since my windows gaming pc was fine. | | |
| ▲ | jeremyjh 16 minutes ago | parent [-] | | We're not really talking about the same things here. I'm talking about the outages where dozens or hundreds of known platforms are down and it makes the front page of HN and major media coverage. That is the magnitude of many of these recent incidents. I know there are more incidents than that. |
| |
| ▲ | tyre 2 hours ago | parent | prev [-] | | There have been many more than two in the past ten years. Maybe not for you, but depending on the region/AZ, especially us-east-1 as it’s aged, outages are impossible to avoid. Much better than Azure, but that’s not a high bar. | | |
| ▲ | jeremyjh 15 minutes ago | parent [-] | | How many made the front page of hacker news and featured on major broadcast networks? Because that is the level of incident we're talking about here. |
|
|
| |
| ▲ | bdcravens 3 hours ago | parent | prev | next [-] | | It absolutely has. The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two) | | |
| ▲ | esseph 3 hours ago | parent [-] | | > The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two) Decades of cycles of layoffs and outsourcing, where the core business or organization no longer operates business-critical or organization-critical functions. | | |
| ▲ | jeremyjh 2 hours ago | parent [-] | | Or, the thing that has happened on a huge scale in much less time than decades. | | |
| ▲ | esseph an hour ago | parent [-] | | It's the combination of moving faster with less domain knowledge, but AI confidence. |
|
|
| |
| ▲ | joesb 2 hours ago | parent | prev [-] | | But how do we know it is because of AI vibe coding though? It could be because some critical infrastructure are due to upgrade after a long delay and this is inevitable at some point. It could be that AI agentic hacking has pushed for some rearchitecture in faster deployment time, which may result in big outage for industry that has so much legacy and used to big bulk deployment only. Or it could just be because we rely more on cloud infrastructure than previously. |
|
|
|
| ▲ | bdcravens 3 hours ago | parent | prev | next [-] |
| Failures aren't just a product of AI-written code in the infrastructure, but the infrastructure trying to bear the load of everyone else's AI-written code at massive scale. |
|
| ▲ | theseamusjames 2 hours ago | parent | prev | next [-] |
| The crowdstrike outage was much much bigger, both in reach and in impact. It took days to weeks to recover from. |
|
| ▲ | CptKriechstrom 2 hours ago | parent | prev | next [-] |
| Remember when AWS messed up their BGP Routes? edit: it was Facebook, not AWS |
|
| ▲ | fourside 3 hours ago | parent | prev [-] |
| But that’s the point around correlation =/= causation. What the parent comment is saying is that the cause could be that attackers have much more sophisticated tools to bring down services. The timing with AI adaption doesn’t prove on its own that lower code quality is the reason. |