Can we at some point acknowledge that constant cloud disruptions are too costly, and can we then finally move all of our hosting back on-prem?

▲

DC-3 10 hours ago | parent | next [-]

It's the old IBM thing. If your website goes down along with everyone else's because of Cloudflare, you shrug and say "nothing we could do, we were following the industry standard". If your website goes down because of on-prem then it's very much your problem and maybe you get to look forward to an exciting debrief with your manager's manager.

▲

lpcvoid 10 hours ago | parent [-]

That's lazy engineering and I don't think we as technical, rational people should make that our way of working. I know the saying, but I disagree with it. My fuckups, my problem, but at least I can avoid fuckups actively if I am in charge.

▲

reassess_blind 10 hours ago | parent | next [-]

How do you mitigate large scale DDoS?

▲

lpcvoid 9 hours ago | parent [-]

I don't, since my stuff is reachable only within the company network/VPN. If I needed to though, I would consult the BSI list of official DDOS mitigation services [0] and evaluate each one before deciding. I would not auto-pick Cloudflare.

[0] (German) https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Si...

▲

reassess_blind 9 hours ago | parent [-]

When the solution you pick inevitably has downtime too you’re in the same boat.

DDoS mitigation is one of those areas that an on-prem solution just isn’t well suited to solve.

	▲	lpcvoid 9 hours ago \| parent \| next [-]
		Yeah, but people aren't using Cloudflare just for DDOS Mitigation. Some are running pretty much everything over it, from DNS to edge caching to load balancing and even hosting. That's what I oppose mainly.
	▲	lousken 7 hours ago \| parent \| prev [-]
		Unless you are really big, onprem stuff would be 90% internal anyway. For everything public you'd host your hardware in a datacenter with better high speed connectivity. And pretty much every single datacenter I interacted with in the last 5 years does have a DDOS protection solution that you can order for your network.

▲

saubeidl 9 hours ago | parent | prev [-]

The problem is the people that sign our checks usually aren't technical, rational people.

The system isn't designed for technical, rational decision making.

	▲	lpcvoid 9 hours ago \| parent [-]
		That's fair, yeah, and I agree it's not always feasible - but if you have any influence over technical direction at your org, I encourage what I wrote above. Otherwise yeah, let the pea counters in the C-Levels dig their own grave.

▲

darkwater 10 hours ago | parent | prev | next [-]

Funnily and ironically enough, I was trying to check out a few things on Ansible Galaxy and... I ended up here trying to submit the link for the CF ongoing incident

▲

rkangel 9 hours ago | parent | prev | next [-]

I would only consider doing stuff on-prem because of services like Cloudflare. You can have some of the global features like edge-caching while also getting the (cost) benefits of on-prem.

▲

donglong 10 hours ago | parent | prev [-]

can you define "constant"

▲

lpcvoid 10 hours ago | parent [-]

Well, between AWS US EAST 1 killing half the internet, and this incident, not even a month passed. Meanwhile, my physical servers don't care and happily serve many people at a cheaper cost than any cloud offer.

▲

alt227 10 hours ago | parent | next [-]

We had an Azure outage in between those 2 as well.

▲

vlovich123 9 hours ago | parent | prev | next [-]

You realize these are two different companies right? If you’re saying “I’m an AWS customer with cloudflare in front” I think you’ve failed to realize that two 99.9% available services in series have a combined availability of ~99.8% - that’s just math.

Your physical servers should have similar issues if you put a CDN in front unless the physical server is able to achieve a 100% uptime (100% * 3 9s = 3 9s). Or you don’t have a CDN but can be trivially knocked offline by the tiniest botnet (or even hitting hacker news front page)

▲

lpcvoid 9 hours ago | parent [-]

I do. But I put both into the "cloud offering off-prem for very much money" shoebox. I setup a CDN once using VPS from different hosting providers for under 100 USD a month, which I would vastly prefer over trusting anything cloud.

And yes, I know that there's sites that need the scale of an operation like Cloudflare or AWS. But 99.9(...)% of pages don't, and people should start realizing that.

	▲	mallets 8 hours ago \| parent [-]
		People who don't need that, also don't care much for an hour or two of service disruption. Most users will have far worse disruptions with the alternatives.

▲

chistev 10 hours ago | parent | prev | next [-]

How do you back up?

	▲	lpcvoid 9 hours ago \| parent [-]
		We have a few colocated servers offsite, each in a different region, each with a zpool of mirrored spinning rust. We use rsync across those at different times.

▲

donglong 10 hours ago | parent | prev [-]

never build on us-east-1, everyone knows that ;)