| ▲ | itzjacki 10 hours ago |
| A colleague of mine just came bursting through my office door in a panic, thinking he brought our site down since this happened just as he made some changes to our Cloudflare config. He was pretty relieved to see this post. |
|
| ▲ | arbuge 8 hours ago | parent | next [-] |
| Tell him it's worse than he thinks. He obviously brought the entire Cloudflare system down. |
| |
| ▲ | mlrtime 8 hours ago | parent [-] | | You joke and I think its funny, but as a junior engineer I would be quite proud if some small change I made was able to take down the mighty Cloudflare. | | |
| ▲ | throwup238 8 hours ago | parent | next [-] | | If I were Cloudflare it would mean an immediate job offer well above market. That junior engineer is either a genius or so lucky that they must be bred by Pierson’s Puppeteers or such a perfect manifestation of a human fuzzer that their skills must be utilized. | | |
| ▲ | anvuong 3 hours ago | parent | next [-] | | This reminds of a friend I had in college. We were assigned to the same group coding an advanced calculator in C. This guy didn't know anything about programming (he was mostly focused on his side biz of selling collector sneakers), so we assigned him to do all the testing, his job was to come up with weird equations and weird but valid way to present them to the calculator. And this dude somehow managed to crash almost all of our iterations except the few last ones. Really put the joke about a programmer, a tester, and a customer walk into a bar into perspective. | | |
| ▲ | jrochkind1 2 hours ago | parent [-] | | I love that he ended up making a very valuable contribution despite not knowing how to program -- other groups would have just been mad at him, had him do nothing, or had him do programming and gotten mad when it was crap or not finished. |
| |
| ▲ | ethmarks 8 hours ago | parent | prev [-] | | A Ringworld reference in the wild? | | |
| ▲ | throwup238 8 hours ago | parent | next [-] | | I never thought I'd get the chance, but then my Claude Code on Web credits ran out and I had to find another way to entertain myself. | | | |
| ▲ | zarathustreal 8 hours ago | parent | prev [-] | | Internet points demand obscure references these days. My system prompt has its own area code |
|
| |
| ▲ | methyl 8 hours ago | parent | prev | next [-] | | I kind of did that back in the days when they released Worker KV, I tried to bulk upload a lot of data and it brought the whole service down, can confirm I was proud :D | |
| ▲ | amalcon 6 hours ago | parent | prev | next [-] | | It's also not exactly the least common way that this sort of huge multi-tenant service goes down. It's only as rare as it is because more or less all of them have had such outages in the past and built generic defenses (e.g. automated testing of customer changes, gradual rollout, automatic rollback, there are others but those are the ones that don't require any further explanation). | |
| ▲ | zidad 8 hours ago | parent | prev | next [-] | | You might want to consider migrating to Azure Front Door if that's a feature you like:
https://www.infoq.com/news/2025/11/azure-afd-control-plane-f... | |
| ▲ | aws_ls 8 hours ago | parent | prev | next [-] | | Well its easy to cause damage by messing up the `rm` command, esp with `-fr` options. So don't take it as a proxy for some great skill which is required to cause damage. | | |
| ▲ | ethmarks 8 hours ago | parent [-] | | You could easily cause great damage to your Cloudflare setup, but CF has measures to prevent random customers deleting stuff from taking down the entire service globally. Unless you have admin access to the entire CF system, you can't really cause much damage with rm. |
| |
| ▲ | 7 hours ago | parent | prev [-] | | [deleted] |
|
|
|
| ▲ | sakisv 10 hours ago | parent | prev | next [-] |
| Well, you can never be sure that he didn't: https://www.fastly.com/blog/summary-of-june-8-outage |
| |
| ▲ | nevf1 8 hours ago | parent | next [-] | | It's also what was the cause of the Azure Front Doors global outage two weeks ago - https://aka.ms/air/YKYN-BWZ "A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions." | |
| ▲ | itzjacki 10 hours ago | parent | prev | next [-] | | Oh don't you worry. We are very much talking about the global outage as if he was the root cause. Like good colleagues :) | | |
| ▲ | rapnie 9 hours ago | parent [-] | | Hmm, wait a minute.. maybe he was the cause! (no, kidding. just upping the pressure as a good peer :) | | |
| |
| ▲ | srmarm 9 hours ago | parent | prev | next [-] | | > May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances. I'd love to know more about what those specific circumstances were! | |
| ▲ | Bloomy22 8 hours ago | parent | prev | next [-] | | I'm pretty sure I crashed Gmail using something weird in its filters. It was a few years ago. Every time I did something specific (I don't remember what), it would freeze and then display a 502 error for a while. | |
| ▲ | CableNinja 9 hours ago | parent | prev [-] | | Damn, imagine being the customer responsible for that, oof | | |
| ▲ | WJW 8 hours ago | parent | next [-] | | What do you imagine would be the result if you brought down cloudflare with a legitimate config update (ie not specifically crafted to trigger known bugs) while not even working for them? If I were the customer "responsible" for this outage, I'd just be annoyed that their software is apparently so fragile. | |
| ▲ | whstl 9 hours ago | parent | prev [-] | | I would be fine if it was my "fault", but I'm sure people in business would find a way to make me suffer. But on a personal level, this is like ordering something at a restaurant and the cook burning the kitchen because they forgot to take out your pizza out of the oven or something. I would be telling it to everyone over beers (but not my boss). | | |
|
|
|
| ▲ | Freak_NL 10 hours ago | parent | prev | next [-] |
| Is there a word for that feeling of relief when someone else fucked up after initially thinking it was you? |
| |
| ▲ | spamizbad 9 hours ago | parent | next [-] | | What’s funny is as I get older this feeling of relief turns more like a feeling of dread. The nice thing about problems that you cause is that you have considerable autonomy to fix them. Cloudflare goes down you’re sitting and waiting for a 3 party to fix something. | | |
| ▲ | mewpmewp2 8 hours ago | parent | next [-] | | Why is it dread? I always feel good when big players mess up, as it makes me feel better about my own mess ups in life previously. | | |
| ▲ | twodave 8 hours ago | parent | next [-] | | Can’t speak for GP but ultimately I’d rather it be my fault or my company’s fault so I have something I can directly do for my customers who can’t use our software. The sense of dread isn’t about failure but feeling empathy for others who might not make payroll on time or whatever because my service that they rely on is down. And the second order effects, like some employee of a customer being unable to make rent or be forced to take out a short term loan or whatever. The fallout from something like this can have an unexpected human cost at times. Thankfully it’s Tuesday, not a critical payroll day for most employees. | | |
| ▲ | mewpmewp2 8 hours ago | parent [-] | | But why does this case specifically matter? What if their system was down due to their WiFi or other layers beyond your software? Would you feel the same as well? What about all the other systems and people suffering elsewhere in the World? | | |
| ▲ | twodave 4 hours ago | parent [-] | | I don't understand what point you're trying to make. Are you suggesting that if I can't feel empathy for everybody at once, or in every one of their circumstances, that I should not feel anything at all for anyone? That's not how anything works. Life (or, as I believe, God) brings us into contact with all kinds of people experiencing different levels of joy and pain. It's natural to empathize with the people you're around, whatever they're feeling. Don't over-complicate it. |
|
| |
| ▲ | kasey_junk 8 hours ago | parent | prev [-] | | Because my customers don’t (and shouldn’t care) it’s a third party. If I caused it there is a chance I can fix it. | | |
| ▲ | _factor 8 hours ago | parent | next [-] | | So you would rather be incompetent than powerless? Choice of third party vendor on client facing services is still on you, so maybe you prefer your incompetence be more direct and tangible? Even still, you should have policies in place to mitigate such eventualities, that way you can focus the incompetence into systematic issues instead. The larger the company, the less acceptable these failures become. Lessons learned is a better excuse for a shake and break startup than an established player that can pay to be secure. At some point, the finger has to be pointed. Personally, I don't dread it pointing elsewhere. Just means I've done my due D and C. | | |
| ▲ | shufflerofrocks 7 hours ago | parent [-] | | Your priority (in this comment atleast) is about the finger-pointing, while the parent's priority is wanting a fix to the issue at hand. |
| |
| ▲ | mewpmewp2 8 hours ago | parent | prev [-] | | If customers expected third party downtime to not affect their thing then you shouldn't have picked a third party provider or spent extra resources on not having a single point of failure? If they were happy with choosing the third party with knowledge of depending on said third party provider, then it was an accepted risk. |
|
| |
| ▲ | sys_64738 8 hours ago | parent | prev [-] | | When others cause problems then you can put your feet up and surf the web waiting for resolution. Oh, wait. |
| |
| ▲ | jspash 10 hours ago | parent | prev | next [-] | | The problem is, I still get the wrong end of the stick when AWS or CF go down! Management doesn't care, understandably. They just want the money to keep coming in. It's hard to convince them that this is a pretty big problem. The only thing that will calm them down a bit is to tell them Twitter is also down. If that doesn't get them, I say ChatGPT is also down. Now NOBODY will get any work done! lol. | | |
| ▲ | hylaride 9 hours ago | parent | next [-] | | This is why you ALWAYS have a proposal ready. I literally had my ass saved by having tickets with reliability/redundancy work clearly laid out with comments by out of touch product/people managers deprioritizing the work after attempts to pull it off the backlog (in one infamous case for a notoriously poorly conceived and expensive failure of a project that haunted us again with lost opportunity cost). The hilarious part of the whole story is that the same PMs and product managers were (and I cannot overemphasize this enough) absolutely militant orthodox agile practitioners with jira. | |
| ▲ | aurareturn 9 hours ago | parent | prev | next [-] | | Every time a major cloud goes down, management tells us why don't we have a backup service that we can switch to. Then I tell them that a bunch of services worth a lot more than us are also down. Do you really want to spend the insane amount of resources to make sure our service stays up when the global internet is down? | | | |
| ▲ | graemep 9 hours ago | parent | prev | next [-] | | Who decided to go with AWS of CF? If its a management decision tell them you need the resources to have a fallback if they want their system to be more reliable than AWS or CF. | |
| ▲ | adriand 9 hours ago | parent | prev [-] | | Haha yeah I just got off the phone and I said, look, either this gets fixed soon or there's going to be news headlines with photographs of giant queues of people milling around in airports. |
| |
| ▲ | jpmonette 10 hours ago | parent | prev | next [-] | | phewphoria | | | |
| ▲ | bookofjoe 9 hours ago | parent | prev | next [-] | | The German word “schadenfreude” means taking pleasure in someone else’s misfortune; enjoyment rather than relief. | | |
| ▲ | bryanrasmussen 8 hours ago | parent [-] | | since schaden is damage and freude is joy, not sure what it should be - maybe Schadeleichtig hmm... | | |
| ▲ | tauchunfall 8 hours ago | parent [-] | | >maybe Schadeleichtig Maybe "Erleichterung" (relief)? But as a German "Schadenserleichterung" (also: notice the "s" between both compound word parts) rather sounds like a reduction of damage (since "Erleichterung" also means mitigation or alleviation). |
|
| |
| ▲ | shortrounddev2 10 hours ago | parent | prev | next [-] | | When I'm debugging something, I'm not usually looking for the solution to the problem; I'm looking for sufficient evidence that I didn't cause the problem. Once I have that, the velocity at which I work slows down | | |
| ▲ | sys_64738 8 hours ago | parent [-] | | My manager once asked if he could have a "quick word". I said "velocity". |
| |
| ▲ | Rooster61 6 hours ago | parent | prev | next [-] | | Schadenfriend? You gain relief, but you don't exactly derive pleasure as it's someone you know that's getting the ass end of the deal | |
| ▲ | 8 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | 8 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | mcphage 9 hours ago | parent | prev | next [-] | | Maybe this isn’t great, but I get a hint of that feeling when I’m on an airplane and hear a baby crying. For a number of years, if I heard a baby crying, it was probably my baby and I had to deal with it. But now my kids are past that phase, so when I hear the crying, after that initial jolt of panic I realize that it isn’t my problem, and that does give me the warm fuzzies. Even though I do feel bad for the baby and their parents. | | |
| ▲ | adriand 9 hours ago | parent | next [-] | | Related situation: you're at a family gathering and everyone has young kids running around. You hear a thump, and then some kid starts screaming. Conversation stops and every parent keenly listens to the screams to try and figure out whose kid just got hurt, then some other parent jumps up - it's not your kid! #phewphoria | |
| ▲ | RhysU 8 hours ago | parent | prev | next [-] | | You're not alone in this feeling. I occasionally smile when it's not my kid. | |
| ▲ | hackeraccount 8 hours ago | parent | prev [-] | | This is one of the secret joys of being a parent. |
| |
| ▲ | 9 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | StanAngeloff 10 hours ago | parent | prev | next [-] | | Schadenfreude | | |
| ▲ | gnfargbl 10 hours ago | parent | next [-] | | Nah, that's delight in someone else's misfortune. This is delight that the misfortune wasn't yours, which is slightly different. | | | |
| ▲ | simonklitj 10 hours ago | parent | prev [-] | | Not quite, that’s more like taking pleasure in the misfortune of someone else. It’s close, but the specific relief bit that it is not _your_ misfortune is not captured | | |
| |
| ▲ | cromka 10 hours ago | parent | prev | next [-] | | vindication? | |
| ▲ | stonecharioteer 9 hours ago | parent | prev [-] | | It's close enough to Schadenfreude but not really. |
|
|
| ▲ | nrhrjrjrjtntbt 9 hours ago | parent | prev | next [-] |
| The company where this colleague works? Cloudflare. |
|
| ▲ | sefke 8 hours ago | parent | prev | next [-] |
| I woke up getting bombarded by multiple clients messages of sites not working, I shitted my pants because I've changed the config just yesterday. When I saw the status message "cloudflare down" I was so relieved. |
|
| ▲ | disconnection 9 hours ago | parent | prev | next [-] |
| Good that he worked it out so quick. I recently spent a day debugging email problems on Railway PaaS, because they silently closed an SMTP port without telling anyone. |
|
| ▲ | 0xblinq 8 hours ago | parent | prev | next [-] |
| Do you guys work at Cloudflare? Do you mind reverting that change just in case? |
|
| ▲ | 0xblinq 8 hours ago | parent | prev | next [-] |
| Plot twist: They work at Cloudflare |
|
| ▲ | raxxorraxor 8 hours ago | parent | prev | next [-] |
| Did your colleague perhaps change the Cloudflare config again right now? Seems to be down again. |
|
| ▲ | bamboozled 10 hours ago | parent | prev | next [-] |
| How do we know your colleagues changes didn't take down Cloudflare though? |
| |
| ▲ | itzjacki 10 hours ago | parent [-] | | Good point. We should probably assume they did, until proven otherwise. | | |
|
|
| ▲ | theoldgreybeard 7 hours ago | parent | prev | next [-] |
| You should tell him his config change took down half the internet. |
|
| ▲ | 10 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | dcjdfvk 8 hours ago | parent | prev | next [-] |
| Even pornhub is down becuase it uses clouflare. |
| |
| ▲ | carlos_rpn 8 hours ago | parent [-] | | Is Cloudflare being down the work of conservative hackers and the rest of the internet is just collateral damage? |
|
|
| ▲ | NitpickLawyer 10 hours ago | parent | prev | next [-] |
| You missed a great opportunity to dead-pan him with something like "No, Bob, not just our site, you brought down the entire Internet, look at this post!" |
|
| ▲ | ants_everywhere 8 hours ago | parent | prev | next [-] |
| Chances are still good that somewhere within Cloudflare someone really did do a global configuration push that brought down the internet. When aliens study humans from this period, their book of fairy tales will include several where a terrible evil was triggered by a config push. |
|
| ▲ | belter 10 hours ago | parent | prev [-] |
| Wait for the post mortem ... It is a technical possibility, race condition propagates one customer config to all nodes... :-) |