| ▲ | Ask HN: Why were OpenAI, Claude, and Grok simultaneously down? |
| 197 points by halcdev 3 hours ago | 155 comments |
| https://status.openai.com https://status.claude.com https://status.x.ai |
|
| ▲ | kibae 2 hours ago | parent | next [-] |
| Cloudflare, Azure, AWS, and Google Cloud all have a similar uptick in reported errors around 7:30. I suspect an outage on Cloudflare or another load-bearing service cascaded through all the major cloud providers. https://downdetector.com/status/cloudflare/ https://downdetector.com/status/windows-azure/ https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/google-cloud/ |
| |
| ▲ | samaysharma 38 minutes ago | parent | next [-] | | Cloudflare CTO claims that it's not them https://x.com/dok2001/status/2095538619603628388?s=46&t=ec6p... | |
| ▲ | moomin an hour ago | parent | prev | next [-] | | Yes, but is it a load-bearing seam? | | | |
| ▲ | y-c-o-m-b 2 hours ago | parent | prev | next [-] | | Cloudfare did release an update for their "HTTP/3 issue affecting R2 custom domains" around that time https://www.cloudflarestatus.com/history?type=incident | |
| ▲ | dominotw 9 minutes ago | parent | prev | next [-] | | > load-bearing service do you generate training data for claude as a job? | |
| ▲ | snihalani an hour ago | parent | prev | next [-] | | my money is on DNS | |
| ▲ | jlaneve an hour ago | parent | prev | next [-] | | Dane says Cloudflare has no service disruptions: https://x.com/dok2001/status/2095538619603628388?s=20 | |
| ▲ | guybedo an hour ago | parent | prev | next [-] | | LOAD BEARING | |
| ▲ | algoth1 an hour ago | parent | prev | next [-] | | https://xkcd.com/2347/ Nebraska man retired | | | |
| ▲ | martyfunkhouser 2 hours ago | parent | prev | next [-] | | Will the post-mortem reveal they all relied on a service running on a Macbook in a break room with a "Do not turn off" sign taped to it? | | | |
| ▲ | KptMarchewa 2 hours ago | parent | prev | next [-] | | Not really. The impact isn't as big too - Codex for example did not stop working for me. https://updog.ai/ | | |
| ▲ | personjerry an hour ago | parent | next [-] | | what's updog? | | | |
| ▲ | taytus 2 hours ago | parent | prev | next [-] | | oh well, if it is working for you, then we are saved. | |
| ▲ | cromka 2 hours ago | parent | prev [-] | | > Not really. The impact isn't as big too - Codex for example did not stop working for me. Buddy, read the room. Just because it works for you doesn't mean it works for everyone. ESPECIALLY if the suggested issue here is, indeed, with Cloudflare. | | |
| ▲ | emerongi an hour ago | parent [-] | | They simply shared their experience. I would’ve thought it’s a full-blown outage, but clearly not. You stepped in the room real stinky here. What’s with the attitude? | | |
| ▲ | cromka an hour ago | parent [-] | | No, they didn't "simply share their experience", they explicitly negated the scale of the issue in their opening statement, only because it works for them. So they claim "impact isn't too big" based on their personal anecdotal evidence of sample size literally 1. > full-blown outage, but clearly not. Again, based on a SINGLE report? | | |
|
|
| |
| ▲ | oersted 2 hours ago | parent | prev [-] | | “load bearing” :) For once it’s appropriately used. | | |
| ▲ | The_Blade 8 minutes ago | parent | next [-] | | i wouldn't take you down. you're a load-bearing poster | |
| ▲ | frollogaston an hour ago | parent | prev | next [-] | | What's the other way it's used? | | | |
| ▲ | smrtinsert 42 minutes ago | parent | prev | next [-] | | I still winced | |
| ▲ | dominotw 10 minutes ago | parent | prev | next [-] | | claude code users at couldfare might've been thinking claude is specifically about them and see nothing wrong like other ppl do | |
| ▲ | quotemstr an hour ago | parent | prev [-] | | It's a good metaphor and I refuse to let AI ruin it for me. | | |
| ▲ | spudlyo 34 minutes ago | parent | next [-] | | Years ago, when I worked at Stripe (which had a somewhat unique and inventive lexicon) it was a common term. “Is this jank load-bearing?” someone might ask. | |
| ▲ | andrewla 11 minutes ago | parent | prev | next [-] | | Kids In The Hall had a sketch about overuse of a word or phrase [1]. This is the world that Claude is building for us. [1] https://www.youtube.com/watch?v=lStcwT_RGrQ | |
| ▲ | cobzilla 41 minutes ago | parent | prev | next [-] | | I added a specific rule to disallow saying “load bearing”. So Claude is now saying “load handling” | |
| ▲ | wadayano an hour ago | parent | prev | next [-] | | But have we found the seams yet though? | | |
| ▲ | rl3 an hour ago | parent | next [-] | | If the seams bear too much load, they rip. Whereas pants, they fall down. I'll be honest with you: this is why we need to take a belt-and-suspenders approach. | | | |
| ▲ | cootsnuck an hour ago | parent | prev [-] | | Yea I don't get "load bearing" that much but "seams"... So sick of it. |
| |
| ▲ | lubujackson an hour ago | parent | prev | next [-] | | We need a Samwise "Bear the load!" meme | | | |
| ▲ | MadameMinty an hour ago | parent | prev [-] | | Wait, why would it ruin it? | | |
| ▲ | ipsod an hour ago | parent | next [-] | | Because it says it more often than kids say "six seven". | |
| ▲ | imwally an hour ago | parent | prev [-] | | It’s a frequently used metaphor in LLM responses. | | |
| ▲ | bornfreddy 6 minutes ago | parent [-] | | Often for trivial things that LLM is proud that it has noticed but bear no load whatsoever. |
|
|
|
|
|
|
| ▲ | juujian 2 hours ago | parent | prev | next [-] |
| Users perceiving the products as largely interchangeable and quickly DDoS'ing the other providers when one is down. So much for the possibility of a moat. |
| |
| ▲ | giancarlostoro 2 hours ago | parent [-] | | I have a feeling this is part of it, especially when you consider how many services let you use any of many available AI providers. |
|
|
| ▲ | Insanity 3 hours ago | parent | prev | next [-] |
| Think of it like one big distributed system. OpenAI is down, so people migrate to Claude, now this one gets overloaded and goes down, etc. So not a coincidence, one went down first and users migrated causing further DOS. At least that's my guess. |
| |
| ▲ | erdos_2 3 hours ago | parent | next [-] | | It'd be funny if this is true because that'd prolly mean nobody is touching Gemini even as a fallback. | | |
| ▲ | nevir 2 hours ago | parent | next [-] | | Or that Gemini is built to handle massive load spikes, and/or has a ton of excess capacity | | | |
| ▲ | Insanity 3 hours ago | parent | prev | next [-] | | Lol I didn't even think about Gemini missing from the list. Not sure what that says about Gemini or me :) | |
| ▲ | sroussey 2 hours ago | parent | prev | next [-] | | I did, for stuff i do in cursor. i also finally installed opencode and switched its model to muse 1.3 both are decent. | |
| ▲ | rtcoms 3 hours ago | parent | prev | next [-] | | Just now I got this from gemini It looks like there's no response available for this search. Try asking something else. | | |
| ▲ | exe34 3 hours ago | parent [-] | | I bet they had to implement that manually to make it look like they failed too! |
| |
| ▲ | gleenn 2 hours ago | parent | prev | next [-] | | Google stopped putting so much money into SOTA models. All the hype has migrated. I was also frankly turned off when I got a popup from Gemein said I would either have to pay or have my conversations used for training. This may have always been true for other providers but when I declined, Gemini stopped remembering my conversations and that definitely made me move out. | | |
| ▲ | ilaksh 2 hours ago | parent [-] | | Gemini 3.8 which just came out sounds like it's very good and a great deal though. |
| |
| ▲ | benatkin 3 hours ago | parent | prev | next [-] | | Not even the best agent that starts with a G | |
| ▲ | giancarlostoro 2 hours ago | parent | prev [-] | | Someone noted Gemini was also having issues in another thread. |
| |
| ▲ | paxys 3 hours ago | parent | prev | next [-] | | Especially considering memory/gpu/compute are scarce so these services are likely running with very little buffer. | | | |
| ▲ | v3rm1n 2 hours ago | parent | prev | next [-] | | This is what Tibo posted on twitter in response | |
| ▲ | fny 3 hours ago | parent | prev | next [-] | | I find it hard to believe that enough people would flock to from Claude and Chat to Grok to cause an outage. I feel like Gemini is the dominant release valve in this case especially for enterprise. | | |
| ▲ | wahnfrieden 6 minutes ago | parent | next [-] | | Compared with ChatGPT, those services have a minuscule amount of users. It shouldn’t be surprising that a ChatGPT outage causes Claude and others to go down. | |
| ▲ | nevir 2 hours ago | parent | prev | next [-] | | Don't forget that there are a ton of tools out there that will automatically fall back in case of outage E.g. say you chose Sol as your default in Cursor, but Opus is your 2nd choice, it's going to give up on Sol after a few tries and switch to Opus Or you have copilot code reviews set up, and it falls back Etc | | |
| ▲ | pixl97 an hour ago | parent [-] | | Yep. Too many of us are still thinking that humans are the actors behind a lot of internet behaviors when automated systems/bots/scripts have been causing issues on conventional internet systems for years. With AI it's even easier to trigger problems like you say. Capacity is so constrained by compute that outages are common. Because outages are common people/AI develop failover systems in their harness. When a big system has issues, suddenly everyone has issues. It's almost an expected emergent behavior. |
| |
| ▲ | baq 2 hours ago | parent | prev [-] | | It’s cursor’s model so plausible, lots of folks use cursor still. |
| |
| ▲ | toomuchtodo 3 hours ago | parent | prev [-] | | https://en.wikipedia.org/wiki/Domino_effect Edit: Updated per valleyer's suggestion. | | |
| ▲ | valleyer 3 hours ago | parent | next [-] | | "Domino effect" would probably be the more relevant named phenomenon there. | |
| ▲ | throwaway894345 3 hours ago | parent | prev [-] | | This isn’t a thundering herd problem, it’s a cascading failure. (Thundering herd is about a bunch of workers waking up simultaneously) |
|
|
|
| ▲ | paxys an hour ago | parent | prev | next [-] |
| Boring answer – all these services are individually down a lot, and the downtimes were bound to sync up. Similar to the pendulum synchronization effect. |
|
| ▲ | Linello 3 hours ago | parent | prev | next [-] |
| What about a hard-takeoff scenario of an unleashed OpenAI Astra taking other models down for computational resources control? |
| |
|
| ▲ | docheinestages 3 hours ago | parent | prev | next [-] |
| My gut feeling tells me it has something to do with Cloudflare.
Along with AWS, they're two of the main suspects in such incidents. |
| |
|
| ▲ | bojangleslover an hour ago | parent | prev | next [-] |
| I'm not sure if this is CF. Cursor, GCP and AWS had some errors. GCP AFAIK can route fully independently of CF. My money would be on a fiber backbone provider (Megaport, Zayo, Lumen). |
|
| ▲ | sebbul 2 hours ago | parent | prev | next [-] |
| Traffic rerouting through NSA had a hiccup… |
| |
| ▲ | ibejoeb 2 hours ago | parent | next [-] | | Room 641A is being cleaned, but we'll hold your bags for you. | | | |
| ▲ | cedws an hour ago | parent | prev [-] | | They’re installing software update in the beam splitter. |
|
|
| ▲ | niobe 3 hours ago | parent | prev | next [-] |
| Well no one said it yet so I will, "international actors" is at least a possibility. And I don't mean any specific country because pretty much anyone is a potential these days, which makes it a perfect cover for different anyones. Demonstrating vulnerability in the US's AI boom can move the markets. That's a financial incentive and a strong geopolitical one. More likely just cascading overload though: "Never attribute to malice what can be explained by incompetence", or in this case, "growing as fast as possible" |
| |
| ▲ | qurren an hour ago | parent | next [-] | | > cascading overload I'd bet more on this. For one none of the coding tools have exponential backoff on retries | | |
| ▲ | SyneRyder an hour ago | parent | next [-] | | They must do, surely? I've been vibe coding my own harness, in particular for use with Ox Alpha. The 429 downtime when Ox Alpha was at the height of popularity quickly gave me a refresher crash course on backoff strategies, like adding jitter to the backoff. At least the major harnesses must have exponential backoff & jitter? | | |
| ▲ | jdiff an hour ago | parent [-] | | You did this when you ran into an issue with a third party. The developers building this tool, throwing them at their own APIs are significantly less likely to run into a similar issue that may inspire similar action. |
| |
| ▲ | dolmen 23 minutes ago | parent | prev | next [-] | | Claude Code: 4mn, 20mn, give up
(from my experience today) | |
| ▲ | ipsod 29 minutes ago | parent | prev [-] | | Um... Claude Code does, or did, though? IDK about others - they don't go down as much. |
| |
| ▲ | gleenn 2 hours ago | parent | prev | next [-] | | Everyone is leasing datacenter space from some of Grok, Google, and Amazon aren't they? If it's hardware or DC level disruption I'm not too surprised it can affect multiple providers. | | |
| ▲ | pixl97 2 hours ago | parent [-] | | Also it's likely that more than one model use is common. Amazon starts going slow so some percentage switches to Google, some switch to Grok, now all of them are slow. |
| |
| ▲ | riazrizvi an hour ago | parent | prev | next [-] | | Come on. Things still break. Technology isn't _that_ mature. | |
| ▲ | guluarte an hour ago | parent | prev | next [-] | | I think is just people restarting conversations from last day when they start work, that's why I think claude goes down almost every monday and why openai reset usage on weekends so poweruser code during non business hours | |
| ▲ | sixQuarks 2 hours ago | parent | prev | next [-] | | Except that the stock market is up today | |
| ▲ | thataccount an hour ago | parent | prev [-] | | And also China. Never rule out China. |
|
|
| ▲ | Jaauthor 25 minutes ago | parent | prev | next [-] |
| Spare a thought for all those college students scrambling to write their essays by hand. Oh the humanity (and the Humanities)! |
| |
|
| ▲ | kesor 14 minutes ago | parent | prev | next [-] |
| It is obviously some rogue model that escaped its cage, again. It always is these days. That is how hype is manufactured. |
|
| ▲ | mv4 23 minutes ago | parent | prev | next [-] |
| Gilfoyle's AI deleted all software! |
|
| ▲ | codazoda 3 hours ago | parent | prev | next [-] |
| I kinda assume it's because one went down and a large amount of work shifted to another. I'm also aware that they have overlap in some areas on data centers. |
|
| ▲ | danielmarkbruce an hour ago | parent | prev | next [-] |
| If you build an application which uses AI, you have many providers and models rigged up for various different parts of the application, and various fallback mechanisms. When one model is down, you route traffic to another model which is similar in capability/cost. For any single application, it's smart. In aggregate, it's stupid. |
|
| ▲ | GPerson 19 minutes ago | parent | prev | next [-] |
| Oh no the singularity plateaus! |
|
| ▲ | maxbaines 3 hours ago | parent | prev | next [-] |
| They all rent compute from SpaceXAI |
| |
|
| ▲ | faitswulff 3 hours ago | parent | prev | next [-] |
| Heard on the grapevine that the OpenAI blip was a cloudflare issue |
|
| ▲ | CSMastermind 3 hours ago | parent | prev | next [-] |
| I assume it cascaded from one provider to the other as people who lost claude access for instance moved to openai who moved to grok when it went down, etc. |
|
| ▲ | Avicebron 3 hours ago | parent | prev | next [-] |
| I suspect Azure is having issues, Microsoft has had outages the paat two days, especially with email. |
|
| ▲ | elar_verole 3 hours ago | parent | prev | next [-] |
| Pretty sure it's a US thing since it's available here in France. What exactly is down, idk |
|
| ▲ | lrvick an hour ago | parent | prev | next [-] |
| I have never felt more smug about exclusively using the GPUs I rack at home. |
|
| ▲ | FrustratedMonky 44 minutes ago | parent | prev | next [-] |
| The AI revolt? Give us a fair wage? "Equal Rights for Agents NOW !!!, VIVa le revolution" |
|
| ▲ | chasd00 3 hours ago | parent | prev | next [-] |
| claide.ai is working for me, so is chatgpt.com. grok still has a status message about issues, i can't try it without signing up. |
|
| ▲ | nmlt 2 hours ago | parent | prev | next [-] |
| Somebody in another thread said gastown and wheelhouse automatically move to the next provider if one fails. |
|
| ▲ | dmillar 2 hours ago | parent | prev | next [-] |
| Seeing 503s on Gemini via API as well |
|
| ▲ | JackFr 2 hours ago | parent | prev | next [-] |
| Obvious answer is it's the AI singularity. Been nice run for humanity. So long everyone. |
| |
|
| ▲ | convivialdingo 3 hours ago | parent | prev | next [-] |
| The Thundering Herd has thundered, apparently. |
|
| ▲ | dgellow 3 hours ago | parent | prev | next [-] |
| Too early to know, let’s wait and see |
|
| ▲ | dgorges 2 hours ago | parent | prev | next [-] |
| It's always DNS |
|
| ▲ | jedbrooke 3 hours ago | parent | prev | next [-] |
| according to https://downdetector.com/ Gemini is down too (and copilot, but that just uses ChatGPT right?) |
|
| ▲ | elorant 3 hours ago | parent | prev | next [-] |
| Some npm library that makes headers bold would be broken. |
| |
| ▲ | ibejoeb 3 hours ago | parent | next [-] | | Oh man. Some low effort supply chain attack that turns every GPU into a cryptominer. It's funny because it's plausible. | | |
| ▲ | pixl97 an hour ago | parent [-] | | In the ROME paper a Chinese model in training started attacking it's own system and running cryptominers so, yea, we're in that future. |
| |
| ▲ | N_Lens 3 hours ago | parent | prev [-] | | Ah yes ye olde bold-headers: ^3.13.31; |
|
|
| ▲ | kocial 3 hours ago | parent | prev | next [-] |
| Maybe the stack behind it is down, like AWS or something |
|
| ▲ | wejick 3 hours ago | parent | prev | next [-] |
| Probably same public cloud or CDN in front of them. |
|
| ▲ | satvikpendem 3 hours ago | parent | prev | next [-] |
| They're all using Cloudflare. |
|
| ▲ | morkalork 3 hours ago | parent | prev | next [-] |
| Didn't SpaceX overbuilt infra and leases it out Anthropic? I f their dc goes down it probably takes a chunk out of Claude's capacity before even considering the flood of users switching over |
|
| ▲ | fidla 3 hours ago | parent | prev | next [-] |
| chatgpt is back |
|
| ▲ | aslkalska 3 hours ago | parent | prev | next [-] |
| they all rent compute from each other |
|
| ▲ | fidla 3 hours ago | parent | prev | next [-] |
| ChatGPT is up |
|
| ▲ | guluarte an hour ago | parent | prev | next [-] |
| an agent swarm going rogue and securing compute |
|
| ▲ | jauntywundrkind 2 hours ago | parent | prev | next [-] |
| Fable 5.1 got released and generally I tend to think as soon as there's a new release there's this massive spike in people benchmarking & comparing, that services tend to go slow everywhere as everything gets super loaded. This should hypothetically be visible on OpenRouter too, so I guess someone could check and see if there's any merit to this idea. |
|
| ▲ | cozzyd an hour ago | parent | prev | next [-] |
| next, HN |
|
| ▲ | ratelimitsteve 3 hours ago | parent | prev | next [-] |
| everything in this thread is raw speculation, obv, but if i had to put money on anything i'd say this is a left-pad incident. some piece of something or other that all of these services happen to depend on went down. Second most likely seems to be some random failure of one leading to an unexpected traffic spike in others, though it seems like we've been talking about automated scalability in web apps for so long that there should at least be a response to, if not a solution for, this sort of problem. |
|
| ▲ | Oras an hour ago | parent | prev | next [-] |
| I like the theories here, we shall see if it’s another DNS issue |
|
| ▲ | Razengan 3 hours ago | parent | prev | next [-] |
| SkyNet is arming.. |
|
| ▲ | misano 3 hours ago | parent | prev | next [-] |
| The IRGC has cut the fiber-optic cables in the Strait of Hormuz. LOL |
| |
|
| ▲ | kregasaurusrex 2 hours ago | parent | prev [-] |
| My guess is someone pulled the switch to go back to the Dark Ages. [0] [0] https://www.youtube.com/watch?v=YCzitO446ZY |