> First of all tons of more outages than ever, remember that React useEffect fkup [0]? Complete insane that this would happen at an infra company that runs a third or so of the web.

Bugs happen all the time. They roughly increase with scale, not decrease. There’s an argument to be made about better testing, but this specific bug seems like a perfect one to slip through: multiple services, hard to spot at code review, involves JS/frontend, invisible at low traffic (test/UT envs).

So IMO it’s not completely insane. Is the implication that Cloudflare should have no bugs whatsoever?

▲

minraws 6 hours ago | parent [-]

Bugs happen all the time yes, outages don't.

If outages are increasing with scale you get 1 or maybe 2 free passes.

After that you either have in-ept Engineering or just in-ept leadership.

I used to be all in on CF a while ago, now I am moving off them almost entirely.

Same issue with GitHub, I can understand if you can't build for the scale when you couldn't predict it but if after over 12-18 months things don't seem to be improving what are you even doing?

I honestly think all of these companies are deluded if they think people will stick around with all these weekly outage events.

I have a homelab server I have had 2 outages in 1 year because my shitty ISP went down. Still at 99.9% uptime, I have done nothing special. I now have backup internet as well.

Is it big? Nope but it doesn't need to be cf scale.

And scale is the reason to use these services why would I use cloudflare if a homelab would have been enough?

If they aren't designing and scaling their systems to handle this scale they might as well close shop, someone else might do it better.

As a infra/dev person who does his own thing on the side, I might be the most impacted by these outages, so I might be coming off as harsh. But they cost me both time/money and headache in extra development work.

Imagine prod deploys are down for 2 days why? Because GitHub actions keep failing...

Oh serving new OTA updates broke? Why? dig into the code.. go oncall with users instead of doing work, realize it's a CF outage and the writes failed. (Feel the tears streaming down your face).

If I have to waste dev time, with AI and me together we could self host it with higher reliability with significantly cheaper costs at this point even at fairly decent scale.

I think any infra company that has more than 1 outage a year is already not worth investing in.

But more than 3 and you might be better off self hosting, even in this ram apocalypse.

If all people in SF are this unserious about reliability (which hasn't been my experience but HN seems especially open to break the prod if you have to)

Then well software companies really do deserve to be replaced by AI.

▲

orf 6 hours ago | parent [-]

To put it bluntly: that is complete nonsense.

AWS has had more than 1 outage a year - is it not worth investing in? Are they not serious?

Outages are just a specific kind of bug, often surfaced by the interactions of several discrete bugs.

Saying “you’re not serious if you have more than 1 bug a year” is silly.

▲

minraws 2 hours ago | parent [-]

This might be rude but this is the reason we have shitty software everywhere. The "there can always be a bug, get over it mindset" is the reason software sucks, everything needs a dozen patches to even work and no one can have any peace in this business.

I honestly don't even want to debate this because I see no point, as honestly the side that would have said outages shouldn't happen is dead buried and out of a job at this point.

But,

> Saying “you’re not serious if you have more than 1 bug a year” is silly.

> Outages are just a specific kind of bug, often surfaced by the interactions of several discrete bugs.

Several bugs or 1 bug? you decide and tell me, when you have made up your mind.

Outages happen as a result of the system's design being shoddy in the first place, and an attitude where bugs shipped are acceptable because it's only a single component, rest of the system shouldn't fail.

It won't go down was the entire thing with microservices and "the cloud" I could link some blogs and brochures if you want me to.

I am honestly over the moon with these conversations on HN, really proves the point why AI is rightly replacing engineers in software, because there wasn't any engineering to begin with.

Imagine if a civil engineer said bridges fall it's fine for a few bridges to fall because bugs happen, and honestly bridge falling is just a kind of bug where several discrete bugs happened at once.

I am certain there is some room to argue about it, but what is clear is that if it was happening every single month you would have stopping using bridges unless absolutely necessary.

I am not sure if there is ever a reply for this honestly, I honestly don't even blame the people just the culture at this point. If that's what software is to some, "something that can fail at any time" and cloud's selling point is just fewer "switches to flip" I am sure I can't change that.

But I won't be deluded into thinking that software outages are just an everyday "bug" and it's fine to have several outages a year, it is the normal mode of operation in-fact no harm done.

	▲	orf an hour ago \| parent [-]
		> Outages happen as a result of the system's design being shoddy in the first place, and an attitude where bugs shipped are acceptable because it's only a single component, rest of the system shouldn't fail. This is correct for some classes of failure but not others. If your authorization system goes down, should you just let any request pass? No. You also need to fail correctly. If we extend the civil engineering analogy, then that would mean fail safely. Two trains derailed in the UK last week. Should we all abandon trains? No? Why do plane crashes happen? That’s the most safety conscious area for software and hardware. Because failures happen. Everyone expects them. Bridges don’t last forever, they know they will fail after a specific time frame and so they design around that. Planes have triplicate systems everywhere. And yet, despite all this, failures still happen. And the fact they know that bridges have specific failure modes and lifespans is the result of a long, long history of bridges failing.