Remix.run Logo
▲ danso 4 hours ago

Why do people assume that this crash is the result of AI vibe coding (i.e. these companies rolling out buggy production code) versus AI-empowered vulnerability seekers? Seems more likely that a group is trying to swarm and exploit a possible vulnerability common to these banks' operations, than it is that Bank of America, Chase, and U.S. Bank all happened to roll out bad vibe code on Monday.

▲codingdave 3 hours ago | parent | next [-]

From the article: "...a technical issue at JPMorgan Chase caused ripple effects throughout the financial system."

Nobody is claiming multiple parallel failures. They are claiming one failure that impacted multiple organizations.

▲bdcravens 4 hours ago | parent | prev | next [-]

Because people want to believe that AI code is inferior, and that it's about to go poof with a return to the previous status quo.

▲JsonDemWitOster 3 hours ago | parent | next [-]

I know the leg I'm standing on is correlation but did we have such widespread outages pre-LLMs? The two that come to mind is that one time some intern pushed a config change that borked us-east-1 and Gitlab deleting their DB. And both had clear comms on the situation within acceptable time frames.

Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason.

▲dghlsakjg 3 hours ago | parent | next [-]

Large outages are not a new thing.

People were making memes about the GitHub unicorn a decade ago.

▲JohnMakin 3 hours ago | parent [-]

No one is saying they are, they’re asking whether the frequency and scale has changed.

▲tadfisher 3 hours ago | parent | next [-]

With GitHub specifically, yes. But I remember AWS/GCP/Azure outages were like a yearly occurrence before the 2020s, and we kept having the same conversations back then about infrastructure resilience, multi-cloud, etc. that were too hard to address then and still too expensive to address now.

▲jeremyjh 3 hours ago | parent [-]

Maybe one major platform had a major outage annually but AWS certainly didn't have a major outage every year. More like every 3-4 years. I remember exactly two in the last 10 years that impacted the production application I run.

▲dghlsakjg 20 minutes ago | parent | next [-]

Here’s a timeline of major AWS outages: https://awsmaniac.com/aws-outages/

Just because an outage doesn’t affect your specific app doesn’t mean it didn’t happen. By that logic Microsoft didn’t have an outage last week since my windows gaming pc was fine.

▲jeremyjh 17 minutes ago | parent [-]

We're not really talking about the same things here. I'm talking about the outages where dozens or hundreds of known platforms are down and it makes the front page of HN and major media coverage. That is the magnitude of many of these recent incidents.

I know there are more incidents than that.

▲tyre 2 hours ago | parent | prev [-]

There have been many more than two in the past ten years. Maybe not for you, but depending on the region/AZ, especially us-east-1 as it’s aged, outages are impossible to avoid.

Much better than Azure, but that’s not a high bar.

▲jeremyjh 16 minutes ago | parent [-]

How many made the front page of hacker news and featured on major broadcast networks? Because that is the level of incident we're talking about here.

▲bdcravens 3 hours ago | parent | prev | next [-]

It absolutely has. The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)

▲esseph 3 hours ago | parent [-]

> The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)

Decades of cycles of layoffs and outsourcing, where the core business or organization no longer operates business-critical or organization-critical functions.

▲jeremyjh 3 hours ago | parent [-]

Or, the thing that has happened on a huge scale in much less time than decades.

▲esseph an hour ago | parent [-]

It's the combination of moving faster with less domain knowledge, but AI confidence.

▲joesb 2 hours ago | parent | prev [-]

But how do we know it is because of AI vibe coding though?

It could be because some critical infrastructure are due to upgrade after a long delay and this is inevitable at some point.

It could be that AI agentic hacking has pushed for some rearchitecture in faster deployment time, which may result in big outage for industry that has so much legacy and used to big bulk deployment only.

Or it could just be because we rely more on cloud infrastructure than previously.

▲bdcravens 3 hours ago | parent | prev | next [-]

Failures aren't just a product of AI-written code in the infrastructure, but the infrastructure trying to bear the load of everyone else's AI-written code at massive scale.

▲theseamusjames 2 hours ago | parent | prev | next [-]

The crowdstrike outage was much much bigger, both in reach and in impact. It took days to weeks to recover from.

▲CptKriechstrom 2 hours ago | parent | prev | next [-]

Remember when AWS messed up their BGP Routes?

edit: it was Facebook, not AWS

▲fourside 3 hours ago | parent | prev [-]

But that’s the point around correlation =/= causation. What the parent comment is saying is that the cause could be that attackers have much more sophisticated tools to bring down services. The timing with AI adaption doesn’t prove on its own that lower code quality is the reason.

▲paimapi 3 hours ago | parent | prev | next [-]

It's not that AI code is inherently inferior, it's that the bulk of the work engineering does now is code review and the scale of reviews has become exponential which means a lot of stuff just gets punted to agents and not carefully examined by anybody with actual subject matter expertise. Plus automation and platform work (like managing test suites) has also similarly become deprioritized and QA in general is being obsoleted as a field (for worse, really).

This is a systemic issue that's driven by AI but the responsibility lies with leadership who are still operating on the move fast and break things mentality. With everybody swinging for the fences and the median blast radius increasing and the checks/roadbumps eliminated you end up with a lot of incidents because the total number of merges and the size of the diff has increased by quite a bit per sprint.

▲jeremyjh 2 hours ago | parent [-]

Also most engineers have always been terrible at reviewing code. They could find most bugs when working through it line by line manually as they wrote the code and the tests, but most could never find bugs in their peer's code through inspection. Now that this - along with business and product understanding - are the primary skills they need, a lot of developers are under-skilled for their job requirements as they suddenly exist today.

Staff-level engineers can do this work because they always have, but if they are suddenly the only ones who can in an org then they are simply overwhelmed.

▲EricDeb 3 hours ago | parent | prev | next [-]

No the issue with AI code is not that it's inferior but that it solves the wrong problem or doesnt believe an edge case is necessary or other contextual issues unrelated to whether the code is actually correct as written.

▲shoobiedoo 2 hours ago | parent | prev | next [-]

I don't know if it's as much as people just hoping that it fails, or the fact that in my opinion a programmer is far more likely to verify code that their reputation might depend on instead of just shoveling in whatever an agent spits out.

▲uludag 3 hours ago | parent | prev | next [-]

Or maybe that cyber attacks are generally known after the fact that they are cyber attacks via things like extortion.

Like take the Binance hack for example. A naughty hacker, instead of stealing bitcoin, could have just caused mass outage and disruption without stealing anything, but didn't because bad actors usually want some financial benefit.

Like if we learn that PII was stolen, then it's be safe to say that this was a cyber attack.

▲lbrito 3 hours ago | parent | prev | next [-]

That's just a strawman argument.

While there may be people arguing for that, the more relevant point is that regardless of any possible positives of AI code, there are clear negatives that are being largely ignored. Specifically in such outage cases, the volume of code being put into production with little or no review or understanding.

▲freecodeio 3 hours ago | parent | prev | next [-]

blaming this problem to vibe coding != ai bad at code

shit programmers have always been a thing, these same shit programmers now putting output from claude straight to production

on top of this there are literally PMs getting involved and doing their own PRs, see vscode recently

▲pessimizer an hour ago | parent | prev | next [-]

> Because people want to believe that AI code is inferior

No, people want to believe that it is good, despite all evidence. However the status quo will not be returning, because it will get better. The catch is that it will somehow end up in your pacemaker or the brakes on your car while it still sucks.

▲rs999gti 2 hours ago | parent | prev | next [-]

AI code is inferior, but business, the media, and regular people don't care about that. They just want fast and cheap, so you get the slop code you pay for.

▲Loughla 2 hours ago | parent [-]

I find it fascinating that anyone thinks computer programs will be different than anything else.

Clothes. Food. Appliances. Tools. Electronics.

All of these things got worse, but got WAY cheaper. All of these industries had to shed excess labor like crazy in pursuit of optimization at the cost of quality.

Why would coding be different?

▲randomNumber7 2 hours ago | parent [-]

Because you write a program once and then you can sell copies without extra costs per unit you sell.

▲whateveracct 2 hours ago | parent | prev | next [-]

if only...

▲Barrin92 3 hours ago | parent | prev | next [-]

>Because people want to believe that AI code is inferior,

This is armchair psychology and extremely bad one because the people who have been concerned with engineering quality have been vastly outnumbered by people who just want to be able to turn their brain off and "ship fast" for probably 15+ years, and that's been reflected in reliability long before LLMs came around.

The code these systems churn out at scale is absolute trash but the financial incentives are to produce out more of it because nobody is ever actually held liable for software breaking in our age

▲fock 2 hours ago | parent [-]

yes, I read some java today where I wished for the clankerd to have done it...

Think system(curl) and random-if-cascades which are never reached - they removed the system(python) in this iteration though...

At this point I think AI-code is just average and probably just fine for corporate environments/SaaS (especially given what thkse produce today), but I still think infrastructure should be a) understood by humans b) produced without installing a backdoor on the dev machine and c) a human should decide what to build and be able to reign in the machine.

▲Grimblewald 3 hours ago | parent | prev [-]

it is though. at minimum qa/QC of ai code is way worse. Things are hitting prod right now in many sectors that should never have hit prod. Amazon on mobile (at least android + ff) is unusable ATM as the most recent example.

▲1970-01-01 3 hours ago | parent | next [-]

Just like humans, it varies wildly. Stating anything other than that is speculation at best.

▲bdcravens 3 hours ago | parent | next [-]

This gets to the implied argument: that humans are inherently better to machine-generated output. The truth is there are a ton of programmers getting paid well to generate crap. Anyone's who been a part of any team that isn't some highly curated A-squad knows this is true. Hell, even teams of 1 know it to be true, when you were rushed, didn't understand the domain or stack, under personal pressure etc.

▲Creamsicle47 3 hours ago | parent | prev [-]

Except human code is reviewable at a reasonable speed

▲swozey 3 hours ago | parent | prev [-]

Before laying off half of my company the last place I worked fired QA first

▲Aperocky 4 hours ago | parent | prev | next [-]

The initial postmortums are out already for the 2 previous event, and they appear to have nothing to do with vulnerability.

▲greenchair 3 hours ago | parent | next [-]

my sweet summer child

▲DEF14A 3 hours ago | parent [-]

What a snarky, condescending thing to say.

▲eliago 2 hours ago | parent [-]

I personally thought it was humorous but I can see the perception.

▲werdnapk 4 hours ago | parent | prev [-]

Do you really believe a guy doing some wiring caused an air traffic shutdown last week?

▲motionlessveloc 4 hours ago | parent | next [-]

Yes, at least in a "straw that broke the camel's back" sense.

Chernobyl was caused by a guy flipping a switch, Challenger blew up because of a faulty O-ring, etc.

▲DaiPlusPlus 4 hours ago | parent | next [-]

That is all true, but those shouldn't be the stated reason that's reported to the public in a press-conference headed up by a US departmental head; that's how misconceptions poison the information-space for everyone.

I also wonder if Trump's administrations' spokespeople even realize they have a massive credibility problem, and that if they want people like me to believe anything then I have to hear it from a mid-level functionary, not a political appointee.

▲pantulis 4 hours ago | parent | next [-]

Proper root cause analysis takes its time so the last obvious fail in the chain of events is what gets published quickly.

▲ 3 hours ago | parent [-]
[deleted]
▲augment_me 4 hours ago | parent | prev | next [-]

You are in a minority, and no-one cares anymore.

You say whatever makes your stocks go up and now down, and if you don't, you are stupid because it's normalized now, there are no repercussions, and if you don't someone else will.

▲ignore 3 hours ago | parent | prev [-]

If the mid-level functionary tacks on, "This is why we demand that Congress allocate $50 billion in new funding for modernization!" do you still believe them?

▲hunterpayne 3 hours ago | parent | prev [-]

"Chernobyl was caused by a guy flipping a switch"

No, just no...Chernobyl was caused by the staff doing an experiment that to put it mildly was wildly risky. They disabled safety systems with wrenches (meaning they couldn't easily be reenabled). Challenger blew up because staff at NASA ignored warnings from the people who made the O-rings. Saying these were "straws" is completely irresponsible. Attitudes like this are what cause outrages. Also, unless you are being dishonest, this is the worst string of outrages in the history of tech. There is a cause. It can be found. And unless you think in these terms, you have no business working with a production system in any way.

▲Liuser 3 hours ago | parent | next [-]

You’re not in disagreement with op. That’s the point motionlessveloc is making, it’s a series of mistakes (straws that make up the hay on the camel’s back) that points to a broader systemic problem.

▲fwip 3 hours ago | parent | prev [-]

I think you missed their point a bit. I believe they are saying that the cause was the faulty wiring in the same way that Chernobyl was caused by a switch flip.

In other words, the wiring problem might be the wrong thing to focus on - but it probably happened, and the systems likely wouldn't have crashed that day without it.

▲DaSHacka 4 hours ago | parent | prev | next [-]

Do you have any reason to believe it didn't?

▲forshaper 3 hours ago | parent | prev | next [-]

It's the sort of thing that is common in all systems I've been inside- which is to say, something little, to do with configuration, hardware fiddling, etc.

▲tygon 4 hours ago | parent | prev | next [-]

While AI rollouts have had issues, it is far from safe to assume just because there is an issue, it is automatically AI-related or vibe code specifically. Things always are breaking. Lines go down, servers have power failures, etc.

▲dclowd9901 3 hours ago | parent | next [-]

With this level of frequency and breadth? The _only_ common factor is AI implementation.

▲tygon 2 hours ago | parent [-]

The trend is your friend until it bites your rear end. To give a topical answer, the Outlook issues were blamed on a software bug from months ago. While AI is being embraced en masse and causing all manner of issues, we have AI issues as _well_ as traditional "it broke" moments. The main point is we should not jump to a conclusion without stated evidence. Just because AI is to blame sometimes, does not mean it always is.

▲cgio 3 hours ago | parent | prev [-]

I think it is the co-occurrence of many incidents across different industries that points to potentially common causality.

▲tiahura 3 hours ago | parent | prev | next [-]

LAX outage is blamed on a single computer

https://www.latimes.com/travel/la-trw-laxdelays15aug15-story...

By Tami Abdollah Aug. 15, 2007 12 AM PT

U.S. Customs officials said Tuesday that they had traced the source of last weekend’s system outage that left 17,000 international passengers stranded in airplanes to a malfunctioning network interface card on a single desktop computer in the Tom Bradley International Terminal at LAX.

The card, which allows computers to connect to a local area network, experienced a partial failure that started about 12:50 p.m. Saturday, slowing down the system, said Jennifer Connors, a chief in the office of field operations for the Customs and Border Protection agency.

As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure a little after 2 p.m., Connors said.

▲BeratnasGas 3 hours ago | parent | next [-]

I automatically trusted the accuracy/honesty of this explanation from 2007. Zero trust today. We have much to fear from fear itself.

▲solyin 3 hours ago | parent | prev [-]

Running Window 98 SE.

▲brendoelfrendo 4 hours ago | parent | prev | next [-]

I can really believe that they plugged in a misconfigured router and that a new router on a faster link was able to flood downstream legacy hardware with junk requests to the point where it started to fail.

▲tiahura 3 hours ago | parent [-]

Are RIP and BGP still a thing?

▲brendoelfrendo 3 minutes ago | parent | next [-]

BGP is very much a thing, and increasingly used inside larger networks instead of just on the edge. RIP should very much be dead, but probably isn't. Afaik OSPF is the interior routing protocol of choice for situations where iBGP isn't used.

▲cestith 3 hours ago | parent | prev [-]

BGP4 is only still a thing on The Internet.

▲mh- 4 hours ago | parent | prev | next [-]

Do I believe a change in networking caused a networking issue? Yes?

▲icepush 4 hours ago | parent | prev | next [-]

I don't know, do computers use wires ???

▲schiffern 4 hours ago | parent | prev | next [-]

  >Do you really believe
Thanks! Conspiratorial thinking hadn't annihilated my last brain cell yet.
▲shermantanktop 4 hours ago | parent | prev | next [-]

I bet he feels bad about it. He should have known that flipping over to the fiber link would take out the whole FAA. /s

▲takeda 4 hours ago | parent | prev [-]

[flagged]

▲ragequittah 3 hours ago | parent [-]

This right here is what needs to stop. AI has its problems but everyone blaming everything on AI is starting to wear thin. By all reports the new SMART AI system wasn't even in any way connected to the network that went down[1]. I guess we can put our tinfoil hat on and say every problem we have from now on is AI's fault and nobody will report it, but I'm not sure that helps anything. Probably better to wait and try to see what actually happened [2] before our knees jerk and credibility gets lost for when actual AI problems occur.

[1] https://apnews.com/article/faa-smart-ai-outage-08af345749c81...

[2] https://www.faa.gov/newsroom/nationwide-ground-stop-question...

▲fwip 3 hours ago | parent | next [-]

Hang on - that first link says: "there’s no indication the communications issues that caused the ground stop are linked to the new computer system,"

As near as I can tell, they are saying that the communication issues are not related to the new computer system (not causative) - not that the new system is physically disconnected from the communications network.

(The article is a bit confusing, using "connected" and "linked to", which could either be describing physical or causative relationships).

▲ragequittah 3 hours ago | parent [-]

FAA's site says:

Q: Was this caused by SMART system? A: No. The two systems are not related. SMART is a data aggregator, and it does not feed information into the air traffic control system.

If you dig a bit further the aggregator gets its data from SWIM, which receives (but does not send) data from the system that went down. I believe it's a 1 way VPN connection that they use to aggregate data because SWIM is consumer facing as well.[1]

So unless something went terribly wrong the AI couldn't have had anything to do with it. But that won't stop it being the top narrative for all time because nobody will look into it this far and we no longer believe any official stories about anything.

[1] https://www.faa.gov/air_traffic/technology/swim/sfdps

▲fwip 2 hours ago | parent [-]

Thanks - I appreciate the clarification. :)

▲Forgeties79 3 hours ago | parent | prev [-]

Can you blame people for not trusting government officials when for the last decade Trump and co have been doing everything in their power to make sure people don’t trust government officials?

▲ragequittah 3 hours ago | parent [-]

Not at all. But the AI would've had to be doing something completely different than its function. It doesn't send or receive data to the system that was overloaded, it's built to receive data from a completely different system. I also think this would be analyzed by enough people that one of network engineers or someone who just generally hates the idea of AI in their workload would've come out and said something by now.

Don't counter lack of credibility by spreading information you can't verify. It just muddies the water even more. There's people out there right now who think it's the AI apocalypse because a banking system is a bit slower.

▲Forgeties79 3 hours ago | parent [-]

I didn’t spread anything. I just think people are inclined to think it’s whatever this current administration says it isn’t. I don’t blame them. There is no shared reality anymore, everyone is basically a little conspiracy theorist

▲randomNumber7 2 hours ago | parent [-]

It will get better at some point. When Trump is away I promise you to blindly trust the government again.

▲fedpost 4 hours ago | parent | prev | next [-]

I don't think the average person cares tbh.

It's caused my AI, it doesn't really matter who's fault it is

▲dclowd9901 3 hours ago | parent | prev | next [-]

The companies in question have confirmed it wasn't due to cyber attacks. That's why.

What's it going to take for the AI pushers to see that maybe this isn't the road to go down.

▲rs999gti 2 hours ago | parent | prev | next [-]

> AI vibe coding

What's new? What has been operating fine until the AI, cloud, and centralized computing came onto the scene?

▲itishappy 4 hours ago | parent | prev | next [-]

I'd assume that whatever happened affected a single centralized service that these banks all relied upon.

▲tedd4u 4 hours ago | parent | prev | next [-]

Like everything else it’s probably a combination of both (as well as other causes).

▲bottlepalm 4 hours ago | parent | prev | next [-]

Everyone is vibe coding now. It's not just you. It's not an assumption.

Vibe coding, vibe reviewing, vibe fixing. No one has a clue how software works anymore.

▲snorkel 4 hours ago | parent [-]

… and no one wants to work anymore! Back in my day we’d hand assemble code on paper and then whistle it over the phone to a 300 baud teletype in the server room … and that shit never broke! Not once!

▲surgical_fire 3 hours ago | parent | prev | next [-]

"AI cannot fail, you can only fail at AI".

Somehow I knew I would find something like this on this comment section.

▲Ancalagon 4 hours ago | parent | prev | next [-]

Why can't it be both?

▲rewgs 2 hours ago | parent | prev | next [-]

IMO the problem is not AI-generated code per se, but rather the utterly absurd expectations and resulting destruction of best-practices and discipline that developers all over are now dealing with.

My first thought for this week's outages has been that people are moving too fast and are, as the saying goes, breaking things.

▲ 4 hours ago | parent | prev | next [-]
[deleted]
▲superfrank 4 hours ago | parent | prev | next [-]

Because there’s a lot of anti-ai sentiment going around right now. Blaming it on AI slop code supports pre-existing beliefs that AI is useless. If it’s AI hackers then that not only means AI is actually good at something, it’s potentially an argument for why we need increased spending and that there really is a race dynamic at play that we need to win.

Basically, blaming AI slop supports a lot of pre-existing beliefs, while blaming AI hackers does not.

▲jenniferhooley 4 hours ago | parent | next [-]

I don't really think it's that. From the people I work with, it's more the push beyond the "capabilities" of AI to the point that we have this amazing tool (AI), which theoretically should allow us to ship much better and more resilient systems. But instead we are pushed to ship really cruddy systems without the proper amount of diligence, review, and testing due to the C-suite's desperation to ship as fast as possible - consequences be damned.

▲retwakm4 4 hours ago | parent | prev [-]

[dead]

▲sunbum 4 hours ago | parent | prev [-]

Hanlons razor. “Never attribute to malice that which is adequately explained by stupidity.”

▲ 4 hours ago | parent | next [-]
[deleted]
▲sanderjd 4 hours ago | parent | prev [-]

I think we need a new razor: "Never attribute to AI incompetence that which is adequately explained by human incompetence."

▲djeastm 3 hours ago | parent | next [-]

"There's three ways to do things. The right way, the wrong way, and the Max Power way!"

"Isn't that the wrong way?"

"Yeah, but faster!"

▲dclowd9901 3 hours ago | parent | prev [-]

If companies are forcing their employees to use AI, which human's incompetence is at play?

▲sanderjd 2 hours ago | parent [-]

Probably multiple humans, I would say!