Remix.run Logo
▲ NoPicklez a day ago

It's impacting all businesses running MS hosted Outlook, that would also include companies with 100,000 of employees. It's a global issues not just one area.

▲stogot a day ago | parent [-]

Well Microsoft made everyone move to managed exchange/outlook right? I was told you can’t self host anymore. How did they not design for regional resiliency?

▲allthatineed 18 hours ago | parent | next [-]

Operational consistency is a hidden single point of failure.

▲NoPicklez a day ago | parent | prev | next [-]

What makes you think they haven't?

Issues that cause outages are complex and can cascade across multiple areas around the world.

Designing for resiliency doesn't mean you won't ever have a major outage.

▲thisisemail 20 hours ago | parent [-]

[dead]

▲EvanAnderson 20 hours ago | parent | prev | next [-]

You can self-host, but it's a monthly subscription instead of a perpetual license.

▲wtch98 18 hours ago | parent [-]

Is there any proof that "self hosted" is still working, wouldn't surprise me if at least part of it relies on centralised services.

In a sane world the CTOs choosing this would be held accountable. The massive global economic fragility which is created by the centralisation of society is a massive risk. Diversity is important, we see it time and time again how optimisation causes panic, from crowdstrike to microsoft. When your outsourced provider is "too big to fail", you have a problem that government needs to deal with.

▲EvanAnderson 12 hours ago | parent [-]

I haven't had occasion to spin-up Exchange SE yet, but I've read plenty of descriptions of other admins running it. I have no reason to believe Exchange itself relies on any centralized features. "New Outlook" has been alleged to use centralized servers even when connected to an on-prem Exchange SE install, and given that it's "modern" MSFT software that wouldn't surprise me, but anybody stupid enough to use "new Outlook" gets what they deserve.

▲Antrikshy 21 hours ago | parent | prev [-]

Well, no one designs to not be resilient. Quality varies in practice.

▲pmontra 19 hours ago | parent | next [-]

My customers are small.

One of them has two servers on AWS behind a load balancer. That's the very least of resilience.

Another one has one server in a datacenter running their primary service on a single VM. That server had issues with RAM chips a couple of years ago that brought it down a few times. However they are not crazy. They made the choice to keep their data at home, bought reportedly good hardware, accepted there there might be consequences. They have other less beefy servers but they did not load balance their service on VMs on different servers.

I think that we can agree that this is no resilience. Is it by design as in "I want my service not to be reliable"? Of course not. Is it lack of quality in the design? Of course it is. But does it matter from the point of view of the users of the service?

▲Symbiote 18 hours ago | parent [-]

I run several services on a VM, with no automation to restore them if there's a hardware issue with the VM host. (There is some resiliency: a managed datacentre, the servers have RAID, dual PSUs, dual network, etc.)

For over 10 years, this has not been a problem. We understand the risk, and know how to restore services onto a different VM host if necessary.

I think the more complex system (e.g. K8S) has problems when people don't fully understand it. We've had outages there, caused by misconfigurations or unexpected behaviour.

▲danaris 12 hours ago | parent | prev | next [-]

While it's probably true that no one specifically seeks to remove resiliency for its own sake, basically the entirety of our corporate world has, over the past several decades, been relentlessly redesigning everything in ways that remove resiliency. Because resiliency is almost inherently antithetical to efficiency. At the most fundamental level, if you want to make sure you have a backup of your data, that means you need at least 2x the amount of storage. And, of course, if you want to make sure you can operate even if a critical employee gets sick/is hit by a bus/goes on vacation, you need to hire extra employees.

And that's anathema to companies nowadays.

▲wtch98 17 hours ago | parent | prev [-]

You design to the needs. I have resilience in my home DNS so I have a 100% availability bounded on the home (if the home is burnt down then I'll lose DNS)

At work I designed DNS to survive the loss of a city, so again 100% uptime. The riskiest part is deploying changes, which is why the deployment doesn't run in parallel and confirms state before deployment.

Resilience isn't hard when you aren't trying to serve a billion customers. The problem is we are addicted to a "winner takes all" economy, which leads to fragility. If Tesco breaks, I can go to Sainsburys, or Aldi, or Lidl, or Iceland, or get in the car and go to Asda or Morrisons. That's diversity.

But in IT the goal is all those follow the same gartner reports and all use the "best supplier" for their backend infrastructure, and if that supplier breaks then they can't restock any supermarkets.

That's a terrible risk and needs to be stopped, not encouraged. But the individual incentives are for the CTO of each supermarket to choose what the consultants tell them. If they don't they are accountable for the choice of not using things like "industry standard".