| ▲ | Software is breaking down at a systemic level(breck-mckye.com) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 67 points by zero_shift 34 minutes ago | 49 comments | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | doug_durham 12 minutes ago | parent | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This narrative isn't consistent with reality. Cloud computing is much older than 10 years ago. We were doing production grade cloud computing over 16 years ago. The OSS revolution dates to the late 90's. The author seems to think that SWE was in a golden age 10 years ago. Sorry, but it's about the same. We have the same problems in SWE as we had 40 years ago. The scale and ambition of the problems we are solving have increased as compute capabilities have increased. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | CharlieDigital 12 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
It doesn't have to be this way. AI is great at following patterns. AI is great at writing tests. AI is great at doing work that humans would consider drudgery and then applying "enough" judgement to it. The problem is that most software teams and most engineers are very, very bad at the thing that AI needs to succeed: technical writing. Documenting the patterns, the rules, the heuristics, the SOP, the judgement; developers could barely be bothered to write comments before (which in the AI age are free, infrastructure-less memory that is always "line-of-sight" for the agent and can be used to enrich with keywords). It's not that AI is the problem, it's that we humans are good at resolving these constraints in the real world at a human pace. AI makes that pace impossible for a human to keep up as a trailing activity. Rather teams need to invest more in technical writing and producing the corpus that AI relies on to learn the operational constraints and rules. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | ljm 7 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Software wasn't exactly hard 12 years ago either, the barriers to entry were being lowered at a rapid pace. 2014 was the dawn of the software bootcamp where JS became more and more pronounced in web architecture. I feel this piece is either more alarmist than it deserves to be, or it is late to the party. It uses various incidents in 2026 to prove a point while dismissing those that did take place in the decade or so previous. For example, heartbleed, the log4j RCE, Rails' mass assignment DOS, Cloudflare updating a WAF rule and breaking the internet, Atlassian deleting almost 1000 of its customers' accounts and data, the Ashley Madison breach, Crowdstrike bricking their customers computers, Okta getting owned multiple times, etc. My own point is that software became like this when we started becoming pragmatic. Nothing to do with AI or management really, we just realised, especially with web tech, that you can release something and then fix it. In the old age of getting CDs or floppies printed and distributed, that was not even remotely feasible. A similar incident in the 90s would involve a product recall or a way to get updated media from a retailer with the fix. Very expensive. Standards have dropped and continue to do so, because we can also fix fast. It's not suddenly breaking down. If anything, it's just too centralised. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | LPisGood 26 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Software has always been broken at a systemic level. It’s a complicated continuously changing web of abstractions and abstraction breaking hacks attempting to survive prolonged contact with reality. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | dividedcomet 26 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This feels a little hyperbolic, the grounding was a result of a cable swap and the ssh cert incident they cite was so human caused it’s not even funny. I think things are a little more shaky, but not massively so. I had the town I was living in during college lose its internet and phone lines due to a cable being cut on a mountain pass and it was fine. If your business is sensitive to an email outage, prioritize high reliability over integration advantages that outlook offers. The php2shell vuln is bad, but we also had log4j years ago. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | spiralpolitik 15 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There have been parts of the software industry that have been mission critical from day one and have built processes, languages, and tools to make sure things are 100% reliable. Building software that way is slower and more expensive and in the past has been an overkill for most systems. The problem software engineers are facing is that now that software is ubiquitous, almost everything needs to be 100% reliable (or close to it). This is coming into conflict with the move fast and break stuff everything needs to be done this quarter at minimal cost mentality. Hence why we are at the breaking point. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | bob1029 6 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
It feels like we are currently experiencing a soft version of the sewing machine riot. Perhaps no one is intentionally destroying anything in this case, but malicious compliance and simple negligence can go a long way when it comes to disruption of complex systems. Destroying a sewing machine requires actual violence. An air traffic control system is an abstract concept. You don't need to touch it to break it. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | anonreplier 22 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There are some decent points, but this is collating a lot of separate incidents with wildly different causes that just happened to occur in a close timeframe. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | skeledrew 7 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Things are breaking down because they were weak to begin with, and demolition is needed in order to rebuild better, stronger. This is just a period of trial by fire: iron has to be heated and beaten numerous times (with carbon) before it becomes steel. All that's happening is AI is accelerating the evolution of industries into something which all eventually have the strength of steel. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | tflinton 8 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
I’ve had AI running to auto upgrade our infrastructure and it’s successful twice as much as humans. Failure rate of a human managed upgrade is 1/10, failure rate of an AI agent is 1/20 in my small set data. Problem is I’ve had 5x more incidents. AI is operating at 10x the speed of a human so even though it’s failing less, it’s causing more incidents per month. I wonder if people are ready for this kind of paradox where using AI may be safer and faster but less reliable for “human time”. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | keithnz 12 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
To talk abstractly about all software, you'd also need to analyze all the software that stays working. This article is just cherry picking a few recent incidents and try's to make some conclusions without really looking at causes and whether that can be generalized to "Software". | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | jstimpfle 20 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
My impression is it's always been a mixed bag. The level of bugginess has been relatively constant. In the meantime, scope and capabilities have gone way up. I hate Teams and Outlook and the virus scanner on my company's laptop though. Those are keeping me from doing my work. Windows + anti-virus makes file access and process spawning 10x to 100x slower compared to a Linux machine. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | tippytippytango 4 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Is defect rate increasing or is the amount of, and use of, software increasing? What is the defect rate relative to that? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | drooby 21 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Most folks here didn't even know, but a Bitrise security outage hobbled our entire CI this weekend. Friday to Sunday night. The security issue motivated all Bitrise customers rotating their secrets. this was a big deal for mobile deployments but didn't get much attention on HN.. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | SoftTalker 17 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> You can find programmers who know how to manage the complexity, but they want to be paid lots of money. This wasn't always the case, by the way. It started with the "dot-com" mania and VC investment in the late 1990s. Before that, a programming job paid decently well but it was on the level of other cost-center professional jobs such as accounting. A comfortable living, but nothing extravagant. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | travisgriggs 15 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> About ten years ago, two small revolutions happened in our industry ... And then, we had the Open Source boom. This seems a bit myopic. Or naive. Open Source "happened" a lot longer than 10 years ago. Microsoft didn't shit itself about Linux just 10 bare years ago. And it's not like prior to 10 years ago, the narrow focus of npm world wasn't cobbling web sites together with bits and pieces garnered from wherever. That said... > But AI is both a cause and a symptom This I agree with 100%. The rise of AI and necessity of AI to continue to make progress is a massive indictment against an industry that prioritized more features and uptime over solid first principles for years, taking out promisory note after note in complexity, in hopes that the stuff papered over would quietly just fade away. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | _pdp_ 5 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
On the flip side, once you hit rock bottom the only way is up. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | chalmovsky 6 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> I believe that tech companies risk losing their “organisational cognition” - their ability to think and respond as a coherent unit. LMAO. They can't lose something they never had in the first place. This article has other pearls like this: > Reddit and Bluesky speculated that even MSFT engineers had no real handle on the complexity of this outage. Like, with all due respect, what the fuck do they know? Also this: >It’s the software that controls your flights, your banking, your petrol pumps, your TV, your payroll. Dude, how are you gonna sneak TV software into the same sentence as flights and banking? :D This is not a serious article, and I don't recommend reading it. But to finish on a positive note, since I've already (regretfully) read it, here's a sentence I like: > This is the real threat: a scenario where nothing in the system is done with intent, and nobody in the system has control. We generate complexity faster than we can tame it. I would just add that this was the case long before AI. And also, just as AI generates complexity at such a great pace, it can just as easily help you fix it and fight that complexity. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | herval 25 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This reads like nostalgia way more than reality | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | rurban 20 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
He obviously is preparing another doomsday conference talk. Nothing interesting to see here. On the other hand software is improving dramatically at a systemic level, even without safe system languages arriving on the horizon. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | titzer 10 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Not everyone is using AI to make shittier software. Most big important software projects I know of are frantically fixing an avalanche of AI-found bugs. Important software is getting more robust and resistant to attacks, while still adding functionality. There's a lot of slop, and it will almost certainly grow quicker than non-slop, but there is a big opportunity for those of us who still work at the bottom to make the pillars more solid, and take on re-engineering projects that would be infeasible otherwise. Case in point: Before Claude a port of my compiler to a new architecture might take anywhere from months to a year. After AI, now that there are multiple backends as examples and an even larger--massive even--test suite, this can take days or less. It took Claude about 3 days to come up with an arm64 backend for Virgil (around April time frame). Then in August it took just 3 hours to build a complete RiscV backend. Both of them pass the entire test suite. And they shook out other bugs elsewhere. Overall, more targets, and more robust. Things can have rocket boosters, you just gotta know where to place them. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | adamddev1 19 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
People keep believing that the big LLM players with somehow disprove Rice's Theorem. "Somehow we'll be able to get AI to check if a program does X or Y." "We'll be able to have the machines verify things automatically." But this is is of course as hopeless as solving the Halting Problem. Theoretically, we know this is impossible. Empirically, we can observe the breaking effects of this broken thinking. Code quality goes down, and apps go from being 99% to 95% reliable. But that's not a big deal, they say, that's good enough... But when the quality of all the building blocks further down get degraded in the same way, you can't keep track of which parts are broken, and which parts are dependable, and the whole thing comes crumbling down. People keep saying, "I can build an app with AI, therefore we can build everything with AI." This article talks about some potential anecdotes that could help to show that doesn't work. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | tcoff91 29 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There were package managers long before npm. Maven, rubygems, etc... | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | AndrewKemendo 20 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Software was mostly always broken but it was hidden and invisible, now it’s visible because the cost of making it visible is so low This is classic surveillance/detection bias: https://healthjournalism.org/glossary-terms/surveillance-bia... It’s basically free to black/grey/whitehat penetration test anything and everything without having to even know what ghidra or metasploit so everything gets penetrated | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | tracerbulletx 17 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
When I first started my career we used to ship like once a month and that was considered good, everything was extremely convoluted and worse than it is today, whole segments of the industry didn't even use Version Control, half the government websites you used stored passwords in the clear and emailed them to you for resets, e-commerce was a cgi script that sent an email for fulfilment and it took weeks to get your shipment, sites went down at night and weekends for maintenance regularly, servers fell over with traffic that would run on a raspberry pi now, but for some reason everyone keeps saying how bad things are now. They're delusional or don't actually remember the past. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | jmclnx 23 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Well there was this saying "Do One Thing and Do It Well", seems once again we are seeing the affects of ignoring that. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | TacticalCoder 14 minutes ago | parent | prev [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
It's because amateurism has been hailed as the second coming of Christ. Leftpad, mentioned in TFA, is a case in point: not because of leftpad itself but because of all those defending leftpadism. "People can't write a proper leftpad". Yeah, well... If a dev cannot write a proper leftpad, then it should never have been a dev in the first place. Someone who cannot write leftpad is not qualified to use leftpad as a dependency (and, no, leftpad is neither the Linux kernel nor a cryptographic libray: it's fucking leftpad, which is the entire point). We've had "devs" whose only qualification was writing glorified "punch the monkey" ads in JavaScript: the JavaScript surveillance/ad complex. Devs who only ever knew the mono-threaded JavaScript ecosystem and suffered from heavy leftpadism. And those very same surveillance/ad complex bros are now generating countless crap with LLMs, in languages they don't understand. They're just mediocre. That's what happened: the veneration of mediocrity. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||