| ▲ | pm90 2 hours ago |
| This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one. |
|
| ▲ | berkes 2 hours ago | parent | next [-] |
| At least this warrants a good review of anyone's dependency on cloudflare. If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...) But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation. But I'm afraid it won't. Just like Microsoft continues to push out software, that, compared to competitors, is unstable, insecure, frustrating to use, lacks features, etc, without it harming their reputation or even bottomlines too much. I'm afraid Cloudflare has a de-facto monopoly (technically: big moat) and can get away with offering poorer quality, for increasing pricing by now. |
| |
| ▲ | zelphirkalt an hour ago | parent | next [-] | | Microsoft's reputation couldn't be much lower at this point, that's their trick. The issue is the uninformed masses being led to use Windows when they buy a computer. They don't even know how much better a system could work, and so they accept whatever is shoved down their throats. | |
| ▲ | rsynnott 30 minutes ago | parent | prev | next [-] | | > Just like Microsoft continues to push out software, that, compared to competitors, is unstable, insecure, frustrating to use, lacks features, etc, without it harming their reputation or even bottomlines too much. Eh.... This is _kind_ of a counterfactual, tho. Like, we are not living in the world where MS did not do that. You could argue that MS was in a good place to be the dominant server and mobile OS vendor, and simply screwed both up through poor planning, poor execution, and (particularly in the case of server stuff) a complete disregard for quality as a concept. I think someone who'd been in a coma since 1999 waking up today would be baffled at how diminished MS is, tbh. In the late 90s, Microsoft practically _was_ computers, with only a bunch of mostly-dying UNIX vendors for competition. And one reasonable lens through which to interpret its current position is that it's basically due to incompetence on Microsoft's part. | |
| ▲ | coffeebeqn 2 hours ago | parent | prev | next [-] | | Vibe infrastructure | | |
| ▲ | rvz an hour ago | parent [-] | | So that is what the best case definition of what "Vibe Engineering" is. |
| |
| ▲ | MrAureliusR an hour ago | parent | prev [-] | | well that's the thing, such a huge number of companies route all their traffic through Cloudflare. This is at least partially because for a long time, there was no other company that could really do what Cloudflare does, especially not at the scales they do. As much as I despise Cloudflare as a company, their blog posts about stopping attacks and such are extremely interesting. The amount of bandwidth their network can absorb is jaw-dropping. I've said to many people/friends that use Cloudflare to look elsewhere. When such a huge percentage of the internet flows through a single provider, and when that provider offers a service that allows them to decrypt all your traffic (if you let them install HTTPS certs for you), not only is that a hugely juicy target for nation-states but the company itself has too much power. But again, what other companies can offer the insane amount of protection they can? |
|
|
| ▲ | jcmfernandes 29 minutes ago | parent | prev | next [-] |
| Absolutely. I wouldn’t be surprised if they turned the heat up a little after the last incident. The result? Even more incidents. |
|
| ▲ | belter an hour ago | parent | prev | next [-] |
| This will be another post-mortem of...config file messed...did not catch...promise to be doing better next....We are sorry. They problem is architectural. |
|
| ▲ | pyuser583 2 hours ago | parent | prev | next [-] |
| Lots of big sites are down |
|
| ▲ | rvz 2 hours ago | parent | prev | next [-] |
| We are now seeing which companies do not consider the third party risk of single point of failures in systems they do not control as part of their infrastructure and what their contingency plan is. It turns out so far, there isn't one. Other than contacting the CEO of Cloudflare rather than switching on a temporary mitigation measure to ensure minimal downtime. Therefore, many engineers at affected companies would have failed their own systems design interviews. |
| |
| ▲ | throwaway42346 an hour ago | parent | next [-] | | Alternative infrastructure costs money, and it's hard to get approval from leadership in many cases. I think many know what the ideal solution looks like, but anything linked to budgets is often out of the engineer's hands. In some cases it is also a valid business decision. If you have 2 hour down time every 5 years, it may not have a significant revenue impact. Most customers think it's too much bother to switch to a competitor anyway, and even if it were simple the competition might not be better. Nobody gets fired for buying IBM The decision was probably made by someone else who moved on to a different company, so they can blame that person. It's only when down time significantly impacts your future ARR (and bonus) that leadership cares (assuming that someone can even prove that they actually lose customers). | |
| ▲ | cryptonym an hour ago | parent | prev | next [-] | | Sometimes it's not worth it. Your plan is just to accept you'll be off for a day or two, while you switch to a competitor. | |
| ▲ | formerly_proven an hour ago | parent | prev [-] | | On the other thread there were comments claiming it’s unknowable what IaaS some SaaS is using, but SaaS vendors need to disclose these things one way or another, e.g. DPAs. Here is for example renders list of subprocessors: https://render.com/security It’s actually fairly easy to know which 3rd party services a SaaS depends on and map these risks. It’s normal due diligence for most companies to do so before contracting a SaaS. |
|
|
| ▲ | karmakurtisaani 2 hours ago | parent | prev | next [-] |
| Probably fired a lot of their best people in the past few years and replaced it with AI. They have a de-facto monopoly, so we'll just accept it and wait patiently until they fix the problem. You know, business as usual in the grift economy. |
| |
| ▲ | 5d41402abc4b an hour ago | parent | next [-] | | >They have a de-facto monopoly On what? There are lots of CDN providers out there. | | |
| ▲ | esseph an hour ago | parent | next [-] | | They do fare more than just CDN. It's the combination of service, features, reach, price, and the integration of it all. | |
| ▲ | immibis an hour ago | parent | prev [-] | | There's only one that lets everyone sign up for free. |
| |
| ▲ | rvz an hour ago | parent | prev [-] | | The "AI agents" are on holiday when an outage like this happens. |
|
|
| ▲ | PlotCitizen 2 hours ago | parent | prev [-] |
| This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers. |
| |
| ▲ | berkes 2 hours ago | parent | next [-] | | But the nature of a CDN and most other products CF offers, is central by nature. If you switch from CF to the next CF competitor, you've not improved this dependency. The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewhere. If you sell, say, petfood online, the dependency-risk that comes with a vendor like CF, quite certainly is less than the investment needed- and risk associted with- building a DDOS protection or CDN on your own; all investment that's not directed to selling more pet-food or get higher margins at doing so. | | |
| ▲ | altmanaltman 44 minutes ago | parent | next [-] | | yeah there is no incentive to do a CDN in house, esp for businesses that are not tech-oriented. And the costs of the occasional outage has not really been higher than the cost of doing it in-house. And I'm sure other CDNs gets outages as well, just CF is so huge everyone gets to know about it and it makes the news | |
| ▲ | agnivade an hour ago | parent | prev [-] | | You can load-balance between CDN vendors as well | | |
| ▲ | otikik an hour ago | parent | next [-] | | Then your load balancer becomes the single point of failure. | |
| ▲ | sofixa 43 minutes ago | parent | prev [-] | | With what? The only (sensible) way is DNS, but then your DNS provider is your SPOF. Amazon used to run 2 DNS providers (separate NS from 2 vendors for all of AWS), but when one failed, there was still a massive outage. |
|
| |
| ▲ | coffeebeqn 2 hours ago | parent | prev | next [-] | | We just love to merge the internet into single points of failure | | |
| ▲ | phatfish an hour ago | parent | next [-] | | This is just how free markets work, on the internet with no "physical" limitations it is simply accelerated. Left alone corporations to rival governments emerge, which are completely unaccountable. At least there is some accountability of governments to the people, depending on your flavour of government. | |
| ▲ | mschuster91 an hour ago | parent | prev [-] | | no one loves the need for CDNs other than maybe video streaming services. the problem is, below a certain scale you can't operate anything on the internet these days without hiding behind a WAF/CDN combo... with the cut-off mark being "we can afford a 24/7 ops team". even if you run a small niche forum no one cares about, all it takes is one disgruntled donghead that you ban to ruin the fun - ddos attacks are cheap and easy to get these days. and on top of that comes the shodan skiddie crowd. some 0day pops up, chances are high someone WILL try it out in less than 60 minutes. hell, look into any web server log, the amount of blind guessing attacks (e.g. /wp-admin/..., /system/login, /user/login) or path traversal attempts is insane. CDN/WAFs are a natural and inevitable outcome of our governments and regulatory agencies not giving a shit about internet security and punishing bad actors. |
| |
| ▲ | inferiorhuman 23 minutes ago | parent | prev | next [-] | | There are many alternatives
Of varying quality depending on the service. Most of the anti-bot/catpcha crap seems to be equivalently obnoxious, but the handful of sites that use PerimeterX… I've basically sworn off DigiKey as a vendor since I keep getting their bullshit "press and hold" nonsense even while logged in.I don't like that we're trending towards a centralized internet, but that's where we are. | |
| ▲ | koakuma-chan 2 hours ago | parent | prev [-] | | My Cloudflare Pages website works fine. |
|