Remix.run Logo
▲ Antrikshy a day ago

Well, no one designs to not be resilient. Quality varies in practice.

▲pmontra a day ago | parent | next [-]

My customers are small.

One of them has two servers on AWS behind a load balancer. That's the very least of resilience.

Another one has one server in a datacenter running their primary service on a single VM. That server had issues with RAM chips a couple of years ago that brought it down a few times. However they are not crazy. They made the choice to keep their data at home, bought reportedly good hardware, accepted there there might be consequences. They have other less beefy servers but they did not load balance their service on VMs on different servers.

I think that we can agree that this is no resilience. Is it by design as in "I want my service not to be reliable"? Of course not. Is it lack of quality in the design? Of course it is. But does it matter from the point of view of the users of the service?

▲Symbiote 21 hours ago | parent [-]

I run several services on a VM, with no automation to restore them if there's a hardware issue with the VM host. (There is some resiliency: a managed datacentre, the servers have RAID, dual PSUs, dual network, etc.)

For over 10 years, this has not been a problem. We understand the risk, and know how to restore services onto a different VM host if necessary.

I think the more complex system (e.g. K8S) has problems when people don't fully understand it. We've had outages there, caused by misconfigurations or unexpected behaviour.

▲danaris 16 hours ago | parent | prev | next [-]

While it's probably true that no one specifically seeks to remove resiliency for its own sake, basically the entirety of our corporate world has, over the past several decades, been relentlessly redesigning everything in ways that remove resiliency. Because resiliency is almost inherently antithetical to efficiency. At the most fundamental level, if you want to make sure you have a backup of your data, that means you need at least 2x the amount of storage. And, of course, if you want to make sure you can operate even if a critical employee gets sick/is hit by a bus/goes on vacation, you need to hire extra employees.

And that's anathema to companies nowadays.

▲wtch98 21 hours ago | parent | prev [-]

You design to the needs. I have resilience in my home DNS so I have a 100% availability bounded on the home (if the home is burnt down then I'll lose DNS)

At work I designed DNS to survive the loss of a city, so again 100% uptime. The riskiest part is deploying changes, which is why the deployment doesn't run in parallel and confirms state before deployment.

Resilience isn't hard when you aren't trying to serve a billion customers. The problem is we are addicted to a "winner takes all" economy, which leads to fragility. If Tesco breaks, I can go to Sainsburys, or Aldi, or Lidl, or Iceland, or get in the car and go to Asda or Morrisons. That's diversity.

But in IT the goal is all those follow the same gartner reports and all use the "best supplier" for their backend infrastructure, and if that supplier breaks then they can't restock any supermarkets.

That's a terrible risk and needs to be stopped, not encouraged. But the individual incentives are for the CTO of each supermarket to choose what the consultants tell them. If they don't they are accountable for the choice of not using things like "industry standard".