Remix.run Logo
▲ nickdothutton a day ago

Microsoft is unable to run a reliable hosted email service, given infinite resources. What people don't realise is that the former is in part a consequence of the latter. Without a forcing function, complexity multiplies, silent fragility accumulates, end-to-end understanding dwindles. There is no incentive to simplify.

▲dv_dt a day ago | parent | next [-]

Im not so sure about the infinite resources part, more like a self inflicted austerity of resources - its a technology problem emergent from a organizational problem, emergent from a certain late stage mindset

▲nickdothutton a day ago | parent | next [-]

~220K staff, ~$330B revenue, ~$200B EBITDA. How they direct those resources (wisely or otherwise, timely or otherwise) is a management decision. I feel I should also point out that when talking about people, "resource" is not the same as "talent".

▲Take8435 a day ago | parent [-]

Are those numbers accurate now that 60k were offered buyouts?

Anyone want to guess if any of those afflicted _could have_ caught an issue that may have prevented this disaster?

▲afavour a day ago | parent | next [-]

1) they haven’t left yet

2) they’re only expecting 10% or so to take the buyouts, so more like 6,000

▲heisenbit a day ago | parent [-]

Now what if those were mythical 10x engineers?

▲dv_dt a day ago | parent [-]

imho, especially in large tech orgs, every engineer is a 10x engineer for the right circumstances. But they're 1x engineers the rest of the time. But the whole reason to keep them around is you get the 10x performance when it's needed. Perhaps handy when a massive outage occurs

▲donny2018 a day ago | parent [-]

Any experienced dev at any org who knows how to deal with that particular pile of mess because it has been their job for at least half a decade is 10x developer.

▲poilcn a day ago | parent | prev [-]

They haven't left the company yet

▲Take8435 a day ago | parent [-]

Do you think those offered buyouts may have less incentive to focus as much?

▲tonyedgecombe a day ago | parent | prev [-]

I have a feeling it's like weight gain. If you run lean your whole life you are fine but if you let the pounds pile on then you are in for a whole lot of pain when you eventually try and lose it.

▲dv_dt a day ago | parent [-]

sometimes you're bulked up because it takes muscles to lift the extra 300lbs of load. now go on a sudden crash diet and see if you can lift the same weight without dropping it

▲gorgoiler a day ago | parent | prev | next [-]

Does having 400M users and 1B emails per day factor into your forgiveness?

I’m no fan Outlook and what they’ve done for the world of email, but if they’re Rolls Royce and this outage is an engine failure then that’s an expected level of service?

Or they’re Boeing and this a 787 Max. I’m on the fence.

▲officeplant a day ago | parent | next [-]

>does having 400M users and 1B emails per day factor into your forgiveness?

Trillion dollar companies shouldn't even be able to have excuses.

You have infinite money, stop fucking up.

▲frolix a day ago | parent [-]

I heard someone threw half a billion dollars at an election and couldn't change people's minds. A company having a bunch of cash doesn't automatically get people to do the right thing. Especially when the priority at every turn is shareholders.

Cash can't fix culture.

▲infamouscow 21 hours ago | parent | next [-]

You're arguing that the governing structure democracy cannot accomplish things with money. That's true, but for completely different reasons. The analogy doesn't really work here because Microsoft is run by a CEO and is functionally a monarchy without any need to persuade anyone beyond their paycheck.

▲soerxpso 18 hours ago | parent | prev | next [-]

Someone else threw the same amount of money in the other direction. Your comparison would make sense if there were a second trillion-dollar company spending their entire budget on trying to make Outlook go down, which is not the case. But we do all know that your goal wasn't actually to make an apt analogy, and just to shove your political whining into an unrelated discussion.

▲ 11 hours ago | parent [-]
[deleted]
▲officeplant a day ago | parent | prev [-]

>I heard someone threw half a billion dollars at an election and couldn't change people's minds.

hard to change the minds of a bunch of pissed off folks when your political campaign funding is pushing AI ads about transfolk and not grocery prices or datacenter rollouts.

▲juujian a day ago | parent | prev | next [-]

Unlike other cloud systems, for email there doesn't seem to be any additional complexities, any dieconomies at scale, no? You could in theory run 100,000 domains as 100,000 separate email services. It's not like social media where everybody is looking at and interacting with everybody's posts. And the email algorithm is well established, it's definitely easier than, say, settling financial transactions.

▲nickdothutton a day ago | parent | prev | next [-]

I don't have very much forgiveness, perhaps only for those junior engineers tending the beast they grew, and which now has indigestion. No-doubt there are all the usual machinations of large corporations at work. Tech companies are no better (in some ways worse) than "normal" companies, because of the invisible complexity of software. Brooks, Ousterhout, and Stroustrup were not fools. But MS run a service, they get freedom to choose how it is engineered internally so long as it continues to present a service interface to the users/client apps/other internal and external services. This is an incredible level of freedom to choose what you engineer for. MS always had the ambition to serve the world with this service. It doesn't seem like they engineered accordingly. It's easy to knock them, but I think they must have made some significant wrong choices.

I'll spare readers the full anecdote, but I worked for the company running the largest mail service in the southern hemisphere, and at the same time a few of the large mail services in the northern hemisphere in the 1990s. We met with the Exchange team who wanted us to adopt their product. Where we could run a given service on 2-3 servers, Exchange sized by MS required 300 of them for just 1 of our countries. Fortunately our cash constraints prevented us going down that road. They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.

▲noisy_boy a day ago | parent [-]

> They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.

Thanks to AI, heated conversations between management and engineering is mostly a thing of the past. Unless the engineer wields an outsized influence, they will be shown the door and replaced with a lower-priced AI-augmented engineer. Which might explain the increasing frequency of shitshows we are seeing lately.

▲moralestapia a day ago | parent | prev | next [-]

>Does having 400M users and 1B emails per day factor into your forgiveness?

No.

Because of the infinite resources premise and because there exist email services of that scale that are reliable. Email is not an unsolved problem.

The rest of your comment does not make much sense to me ...

▲steveBK123 a day ago | parent | prev | next [-]

No, and I've been discussing this with a friend today.

Internal software goes through rigorous change control, change windows, change freezes around key business dates, etc to minimize risk.

Then you sign up for some SaaS and they blow out email (which aside from security needs 0 updates lets be honest) midweek, mid business day without warning or reason.

▲oddIdentities a day ago | parent | prev | next [-]

>1B emails per day

Jesus, make sure you stay away from signal processing if this trickle of data scares you

Just for a sense of scale, on a single CPU core that is something like 1M instructions per email. ONE CPU CORE.

▲diegolas a day ago | parent [-]

and it's all done automatically by a dedicated device, what's your point?

▲apexalpha a day ago | parent | prev | next [-]

They could've split it into smaller e-mail services in stead of designing one monolith for the planet.

I mean it's e-mail.

▲tonyedgecombe a day ago | parent | next [-]

It's probably hacked together from bits of the on-prem version. We all know what these places are like on the inside. It's not the smooth running engine they like to present it as.

▲Caloric3002Swed a day ago | parent | prev | next [-]

Its not "email"

Its all connected

M365 is a suite. Make a 365 email and you also make an account in Entra, connections to onedrive, sharepoint, teams etc etc etc

Its not "just email"

▲jscd a day ago | parent | next [-]

I was digging through Teams API calls the other day and found some references to Skype in the responses. It’s all connected indeed

▲basch 19 hours ago | parent [-]

Skype for Business was previously Lync, which is what the teams voip components are based on. (Teams is essentially a lync/sharepoint client.)

They just one day decided to rename it Skype for Business while making it incompatible with Skype because brand synergy.

▲Khaine 17 hours ago | parent | prev [-]

Its also a giant pile of shit.

Teams - absolute garbage OneDrive - clunky garbage Sharepoint - massive pile of shit

▲NigeNigeNige a day ago | parent | prev [-]

Every service becomes a monolith if it becomes popular enough.

▲stogot a day ago | parent | prev [-]

When Microsoft has an outage it tends to be global. They don’t seem to learn institutionally how fault tolerance should work across a planet. All else- I wait for the postmortem to critiq

▲zkmon a day ago | parent | prev | next [-]

Great insight. I would add that forcing function comes from the self, not from outside. The force is essentially the communication between the parts. When the communication breaks down, things fall apart, establishing a new kind of communication across the pieces to achieve equilibrium. The physical structures also have the same rule. They will crumble into a pile of lego bricks to achieve a new state of equilibrium and a simplified network of communication across the pile.

▲ 17 hours ago | parent | prev | next [-]
[deleted]
▲torben-friis a day ago | parent | prev | next [-]

Today is sounding like one hell of an incentive.

▲silvestrov a day ago | parent | next [-]

Why?

Do they lose any significant number of customers? Do anybody in leadership get fired?

Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?

Or do it sound more like an organization that is optimized for evaporating responsibility?

I think this will change nothing.

▲torben-friis a day ago | parent | next [-]

Well, at the very least there's going to be a lot of people pointing at their broken SLAs.

>Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?

Not implying otherwise, to be clear. Very far from defending MS.

▲dnikolovv a day ago | parent | next [-]

Pointing at SLAs does nothing. Maybe they'll issue a discount or two. Getting rid of Microsoft in a corporate setting is close to impossible.

▲vladms a day ago | parent | next [-]

Many things change if the next generations will select another service because they have a bad impression (correct or not) about something.

Did you personally select anything IBM lately? At some point many years ago, people were saying "Getting rid of IBM in a corporate setting is close to impossible.

▲zkmon a day ago | parent [-]

Actually it is about the perceived risk distribution within an org, not really about the quality of the product. "No one got fired for buying an IBM" directly tells the decision maker that their risk of getting questioned internally is low due to the brand reputation of IBM. It's a play on risk.

▲brunoarueira a day ago | parent | prev [-]

The main problem to get rid of Microsoft, is how they sold their services in the past more related to regulated environments. I worked at a place where we're using Google tools but with AWS as cloud provider, then one time we need to decide which tools we should continue use, the choices were Google and Microsoft. Microsoft won because they already had at the time the better auditable tools to track any single event across them which Google lack up to a certain point, we kept at least AWS though

▲ssss11 a day ago | parent | prev [-]

I wonder if clients will take to court for damages/liability?

▲netdevphoenix a day ago | parent | next [-]

I imagine there is some kind of arbitration included in the service

▲UqWBcuFx6NV4r a day ago | parent | prev [-]

No. That is contracted out.

▲steve1977 a day ago | parent | prev [-]

And why would it? Most people using Microsoft 365 have bullshit jobs anyway. All they do is performative. They are not productive, with or without access to their e-mail and calendar. It doesn't make a real difference.

▲UqWBcuFx6NV4r a day ago | parent [-]

let me guess, the only real jobs are blue collar ones, and software developers of course, or whatever it is that you happen to do. May as well comment “I’m a bit green behind the ears” and leave it at that.

▲steve1977 a day ago | parent | next [-]

If anything, it's the opposite. Too many decades in corporate environments to not become cynical. Too many instances of having seen what the man behind the curtain is actually doing.

And I don't exclude myself here by the way. I know what I'm doing, I know what my colleagues are doing, I know what my managers are doing. We're all pretty busy, but our productivity is very low.

▲oddIdentities a day ago | parent | prev [-]

how do you figure?

▲bigfudge a day ago | parent | prev | next [-]

Seriously, who will change provider because of this? Our institution is so brainwashed and hidebound to Microsoft it would take a lot more than this. It seems like an abusive relationship: the bad times just increase fear generally, so make change less likely.

▲throwawaybackag a day ago | parent | next [-]

It's the same with AWS, Cloudflare. There is also this weird effect, often commented on here, where if half the Internet is down, everyone understands when you are down. If I use some random new start up for my infrastructure or services, and it goes down, customers and executives are furious. So it seems we are stuck here.

▲smackeyacky a day ago | parent [-]

Microsoft is worse. At least you can entertain the idea of switching between AWS and GCS but moving off outlook or Entra? Not a chance

▲cutler a day ago | parent [-]

What's wrong with GMail?

▲XajniN a day ago | parent [-]

[dead]

▲eterm a day ago | parent | prev | next [-]

It might not force a change away, but it definitely reduces the chance of people switching to them.

▲hobofan a day ago | parent | next [-]

I think what it will mostly do is hardening existing customers anti-cloud stance. There already was big reluctance by many enterprises to switch to Exchange Online and New Outlook due to lack of feature parity and control, to the degree that Exchange Server SE became a product (with support until at least 2035) and Classic Outlook is still the prescribed Outlook variant in many organizations (with support until at least 2029). I also know of some customers, which migrated back from Exchange Online to on-premise Exchange Server because they were so unsatisfied with it.

This will only make it harder for Microsoft to pull all customers to their cloud offerings. I think in general that's a significant blow to SaaS-ification of the field.

▲Geezus_42 a day ago | parent | prev [-]

Doubtful.

▲tonyedgecombe a day ago | parent | prev [-]

I suspect people will take it in their stride. What would happen after a month long outage is another question.

▲41353aab a day ago | parent | prev [-]

[dead]

▲gabucino a day ago | parent | prev [-]

tl;dr

https://en.wikipedia.org/wiki/XBill