That’s what I also thought when I saw this incident. I wonder if there’s something up internally at Cloudflare or that it was always like this.

I feel like something such as a route leak should not be something that happens to Cloudflare. I’m surprised they set their systems up to allow this human error.

▲

jacquesm 4 hours ago | parent [-]

John left in April last year I think so it probably isn't directly related, so please take my comment in jest, but still it is worrisome, CF is in many ways 'too big to fail' and if this really becomes a regular thing it is going to cause a lot of people focused on their 'nines' to be pissed off.

One thing to their credit though: BGP is full of complexity and it definitely isn't the first time that something like this goes wrong, it is just that at CF scale the impact is massive so there is no room for fuckups. But doing this sort of thing right 100% of the time is a really hard problem, and I'm happy I'm not in any way responsible for systems this important.

Whoever is responsible learned a lot of valuable lessons today (you hope).

▲

rkagerer 2 hours ago | parent [-]

The older I get, the less I buy into "too big to fail" arguments. I now view it as "can't fail soon enough". The sooner it breaks down, the sooner something better will supplant it.

This last sentiment holds true generally since organizations no longer subject to meaningful competition inevitably squat on their laurels and stop excelling at the things they used to be good at. We've seen it everywhere - Boeing, Google, Microsoft (with OS's), etc.

	▲	mschuster91 2 hours ago \| parent [-]
		> The sooner it breaks down, the sooner something better will supplant it. That's not always possible, because the counterparty - aka threat actors - is always growing bigger, and you practically need to be the size of Cloudflare, Akamai or the Big 3 cloud providers to be able to weather attacks. You need to have big enough pipes to data centers and exchange points worldwide, otherwise any sufficiently motivated attacker can just go and swamp them, but big pipes are helluvalot expensive so you need to have enough large and financially capable customers. That's also why Cloudflare has expanded their offerings so much (e.g. Zero Trust), they need to have their infrastructure at some base load to economically justify it. And that's also why Cloudflare will not be kicked off the throne any time soon. First of all, the initial costs to set up a competitor are absurdly high, second, how is a competitor supposed to lure large long term customers away from CF? Any case, the real "fix" to Cloudflare being too-big-to-fail isn't building up competitors, it's getting the bad actors off of the Internet. Obviously that means holding both enemy (NK, Russia, China) and frenemy (India, Turkey) nations accountable, but it also means cleaning up shop at home - the aforementioned nation states and their botnet operators rely on an armada of hacked servers, ordinary computers and IoT devices in Western countries to carry out the actual work. And we clearly don't do anywhere near enough to get rid of these. I 'member a time when writing an abuse@ mail report that this would be taken seriously and the offender being disconnected by their ISP. These days, no one gives a fuck.