Remix.run Logo
▲ Ungh a day ago

I've had two ex-colleagues contact me this morning in UK government positions and tell me that they're still out of action with no plan B in place.

I hope this sort of stuff pushes us to a distributed model rather than a centralised mono-provider one.

▲RyanHamilton a day ago | parent | next [-]

Crowdstrike crashed 8 million+ machines and broke many so bad they didn't fix on restart. Their stock is up 3x since then: https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou... . I don't understand it but I also wouldn't bet on anything changing from this outage.

▲ifwinterco a day ago | parent | next [-]

Businesses rationally won't switch because of one outage (even a bad one) because switching costs for a lot of places will be astronomical.

So again rationally this shouldn't affect the stock price much.

However repeated disruptive outages every few weeks/months (à la GitHub) is a different story, and there's also the invisible effect of businesses maybe being less likely to choose Microsoft in the future

▲rifkiamil a day ago | parent | next [-]

It must be 15–20 years now that Mimecast has been offering a protective layer around email services like Microsoft Exchange and Microsoft 365. Even today, you can see companies that are still able to keep their email running during Microsoft outages because they have Mimecast's email continuity service in place. It shows the value of having that extra layer of resilience.

▲rzerowan a day ago | parent | prev | next [-]

If the downtime persists for more than 72 hours some will switch as they need to conduct their business , whether its email/github etc. However as long as the outages are more like a snowday very few of the big orgs will have the inclination to get back into the business of self-hosting for accountability/price/biz inertia reasons.

A week long outage though , yeah at that point the decisions becomes a nobrainer and not even network effects will prevent the migration.

▲NetMageSCW a day ago | parent [-]

Even 72 hours isn’t enough - how will they conduct business with anyone else if they are still down and how will they get their old emails to resume conversations and how will they transfer their known email addresses or inform others of their new email addresses? None of that is cheap or easy.

▲nagaiaida 18 hours ago | parent [-]

what big businesses use provider-domain email addresses that would necessitate new email addresses just for switching mailserver hosting?

▲freehorse a day ago | parent | prev | next [-]

Regular businesses sure, but governments, public and essential services should have different criteria imo and should rethink their reliance on these services.

▲newsclues a day ago | parent | prev | next [-]

Having an outage, fixing it, and then not having outages in the future, may be a positive signal compared to no outage ever, because you assume it will happen and the recovery time and process is unknown.

▲wickedsight a day ago | parent | prev [-]

> switching costs for a lot of places will be astronomical

Switching to what though? It's not like we didn't have outages before SaaS. It just happened at different times throughout the year in stead of all having an outage at the same time.

I feel like the big difference right now is that management can't call the EoD for progress updates every 10 minutes, they just have to assume that someone at Microsoft is working on fixing the issue.

▲ifwinterco a day ago | parent [-]

Indeed, realistically in 2026 switching from Microsoft to Gsuite, those are the only two things I've ever seen companies use, and Google doesn't have 100% uptime either.

So yes that's another argument to not do anything and stick with Microsoft unless things become terrible

▲Ungh a day ago | parent | prev [-]

Yeah true. Only by heel dragging and refusing to roll it into our production stuff lead to us surviving that.

▲roca a day ago | parent | prev | next [-]

The reality for most companies, up until now at least, is that if they manage email servers themselves they will have more annualized downtime than if Microsoft does it. Of course if this outage lasts long enough (who knows what's going on over there), that will stop being true.

IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.

Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.

There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.

▲Ungh a day ago | parent | next [-]

That's really not true though. That's what the cloud companies told us.

We've had more downtime from all our SaaS and cloud services in the last year than the 20 years before when self hosting.

▲doubled112 a day ago | parent | next [-]

When we self hosted, we very rarely had an unplanned outage. It really shouldn’t happen all the time. Plus I could work planned outages around the company’s schedule.

Now it feels like something is always down.

▲venzaspa a day ago | parent | prev [-]

That's all predicated on having a team of people with the appropriate knowledge of how to securely self host an email system. Some organisations aren't big enough for that to be worthwhile, others have decided to forego that cost to let the cloud providers pick it up.

Email is not a particularly easy thing to reliably host and secure for a sizable organisation, especially if its not your main focus.

▲fuussssip a day ago | parent | prev [-]

Annualized outage minutes is the wrong metric. If my company's email is down for fifteen minutes, twice a day, at 2am and 3am, that's over a week of downtime. But we might not even care.

If I lose a whole day, we lose all the productivity for that day and delay countless upstream and downstream efforts by a day.

The distribution of those minutes matters.

▲roca 19 hours ago | parent [-]

That's true, but it's more true for some companies than others. A lot of big companies cross time zones.

I suppose "unplanned outage minutes" would be a better metric. I still expect most companies to be worse on that metric. Most companies are just bad at running services reliably, especially when it's not their core business.

▲gpvos a day ago | parent | prev | next [-]

Maybe use a decentralized protocol, something like SMTP...

▲pwndByDeath a day ago | parent | next [-]

LXMF FTW

▲Ungh a day ago | parent | prev [-]

All good until you run it all in the same business :)

Rather smug with my separate SMTP/IMAP provider now. And hourly offline backups of the mailboxes.

▲sosodev a day ago | parent | prev | next [-]

My small US gov org has been pushing more of our infra into centralized Microsoft services. The rationality seems to be that outages would be worse if we managed our own stuff. Nobody trusts each other and choosing Microsoft is an easy way to shift the blame outside the org. Nobody ever gets fired for choosing Microsoft and any issues they have must be justified.

▲AdamN a day ago | parent | prev | next [-]

It's ironic that the Internet was designed to accommodate nuclear war and we still decided to consolidate so much critical infrastructure onto monoliths.

▲mottalli a day ago | parent [-]

Internet is working perfectly fine, it's the services that run on top of the Internet that are failing

▲antiframe a day ago | parent [-]

SMTP is a protocol designed back when machines might connect once a day to exchange email. It was designed to work during intermittent connectivity, queue messages, and recover gracefully. We instead decided we like instant rather than reliable by putting all our emails in one basket and then forgot that it doesn't have to be that way.

▲sobiolite a day ago | parent | prev | next [-]

It should, but knowing UK gov they'll instead pay Accenture and IBM £10bn for a backup system that'll take 15 years and never get delivered.

▲Ungh a day ago | parent | next [-]

It's usually worse. They spend it in many £500k consultancy blocks asking how to fix something, get a report back, don't implement it and then the staff who were working on it get a position in another part of the government.

▲wowczarek a day ago | parent [-]

Beat me to it. But not before the National Email Resilience Review Committee is formed first, chaired by Lord Stoate Coate, announced with great fanfare. Obviously separate from the 1300 existing general cyber resilience review committees. Now then, very well, chaps and what have you, carry on, yees? (rolling Rs necessarily).

▲Borg3 a day ago | parent | prev [-]

Exacly, because 10B is big enough that everyone involved can get a bit of slice from it... Corruption is astronomical these days.

▲MisterMunchkin a day ago | parent | prev [-]

I mean why is outlook even centralised? If a company has 600,000 employees, why don’t they get their own infrastructure? Why is it all dependent on central Microslop infrastructure in America

▲dijit a day ago | parent | next [-]

Because microsoft sort of pushes you into it.

▲NetMageSCW a day ago | parent | prev | next [-]

Because there’s more to Microsoft 365 than email and there are no good solutions for the non-email parts.

▲rifkiamil a day ago | parent | prev | next [-]

It's on days like this that Microsoft could really learn a thing or two from Google's SRE culture. These large tech companies have far too many internal dependencies, creating massive blast radii when something goes wrong.

▲pjc50 a day ago | parent | prev | next [-]

It's a pain to administer.

▲Leonard_of_Q a day ago | parent | next [-]

Not nearly as much pain as those who stand to benefit from outsourcing mail services would like you to believe.

▲pjc50 a day ago | parent [-]

I have administered both Exim and Exchange and would very much like not to do so again. Especially since in the time since then the number of requirements for effective delivery and antispam has only gone up.

▲TheOtherHobbes a day ago | parent | prev | next [-]

Paying employees -> bad. Paying corporations -> good.

▲OccamsMirror a day ago | parent | prev [-]

As if this isn't solvable with automation.

▲Ungh a day ago | parent | prev [-]

That is exactly the question that needs asking and answering.