| ▲ | BLKNSLVR a day ago |
| How many Microsoft employees are working on this? It's concerning that one of the biggest, most valuable companies on the planet, that runs services internationally, can have one of its primary services have gone down for this long. 6 hours ago they seemed to not exactly understand what the problem was, and that was 6 hours into the outage. This is concerning. |
|
| ▲ | sobiolite a day ago | parent | next [-] |
| Not so long ago, I was assigned to a project that had to be built on Azure due to client requirements. Because we were underhanded and it was an MVP, we'd elected to use PaaS as much as possible: Postgres Flexible Server, Static Web Apps, Easy Auth, etc. On paper, these all looked fine. They had lots of documentation, example projects, GitHub repos, blogs, roadmaps, etc. Then we started building and running into issues, and we realised a bunch of these offerings were basically unmaintained. There were features that had been stuck in "preview" status for literally years. The blogs and roadmaps were no longer updated, and issues were piling up in GitHub. I started digging into the historical blog posts, GitHub comments, etc. and I realised everybody at MS who'd been working on them, SWEs, PMs, architects, etc. had either left the company, or moved to other teams. And the one constant about the internal moves was that people were now working in some "AI" related project or product. Literally, I didn't find a single person who wasn't now working on AI. Clearly, there has been a huge internal push to get into AI. Resources have been reallocated, and anything non-AI related has been starved. And since working on an AI project is probably considered the only route to advancing your career, anybody with any ambition has moved to a team doing something AI-related. The result is that huge numbers of products and teams doing vital but non-AI related work have been hollowed out, and now we're seeing the consequences. |
| |
| ▲ | mjb8086 a day ago | parent | next [-] | | This state of Azure saddens me. I learnt everything cloud with it. I've built many things (big and small) in it. It was the one part of Microsoft I thought they'd not neglect. Lots of its features - App Services, DNS, managed DB, Front Door, Blobs, Durable Functions, Service Bus, App Insights - are very powerful. The frontend portal is mostly fine for management. The Az CLI makes it easy to provision things quickly. Coupled with .NET Core on MacOS, I've not needed to touch Windows locally for years. I could learn AWS or GCP, but that's just going to cost more time when I'd rather use what I already know to build more things. With Azure, I'd hate to see the rot set in further. | |
| ▲ | mrngld a day ago | parent | prev | next [-] | | That's related to something I realized which I think a lot of people might find amazing. They think of companies like Apple and Microsoft as trillion-dollar, globe-spanning companies, so it must be massive teams that work together on everything. But no, read Hacker News or other niche places enough and eventually you realize a lot of the things we all know or use was a solo effort or a very small number of people, sometimes not even an assigned project but something they did on the side. Cue the xkcd about the lone open source project maintainer holding up the entire modern web but apply it to the tech giants and there you go, it happens there too. | |
| ▲ | cryptonym a day ago | parent | prev [-] | | https://news.ycombinator.com/item?id=50250937 |
|
|
| ▲ | pjmlp a day ago | parent | prev | next [-] |
| Let me put this way, the Visual C++ standard library has officially one FTE working on it, everything else are open source contributions. WinUI 3.0, WinAppSDK, and related CsWinRT, C++/WinRT are in a ridiculous position six years after Project Reunion announcement, due to ever meagre resources. XBox app keeps under delivering user experience, and bugs on patch Tuesdays are getting a joke by now. All this in a four trillion valued company. |
| |
| ▲ | ctmlr a day ago | parent | next [-] | | Maybe acting like jerk-offs got them to four trillion but there are consequences to burning all your social capital. A large camp of software developers have genuinely hostile feelings towards Microsoft, I would bet that this affects the quality of talent they can hire and these are downstream affects. Lately it seems like more non-techies are souring on the company. | | |
| ▲ | sznio a day ago | parent | next [-] | | >A large camp of software developers have genuinely hostile feelings towards Microsoft which, besides hatred, manifests in malice, like Nightmare Eclipse. I wouldn't be surprised if this outage was an engineer leaving a parting gift, using a bug they probably knew of for years but management didn't allow them to fix. | | |
| ▲ | bix6 a day ago | parent | next [-] | | Wow Nightmare Eclipse is fascinating. It sounds like he is operating in a gray area and could be charged if Germany really wanted to? | |
| ▲ | aslihana a day ago | parent | prev | next [-] | | Agree. I was having suspicions around that | |
| ▲ | a day ago | parent | prev [-] | | [deleted] |
| |
| ▲ | fifticon a day ago | parent | prev | next [-] | | Tellingly, I will not go to MS for UI stuff.
When I have to anyway, I use their decades-old APIs.
It is a canary with regards to microsoft,
that they have no known champion for their UI stuff,
no trusted face for their UI stuff;
the company all just appears to coast along on
"I assume someone is handling it?"
The main microsoft employee I trust is Raymond Chen,
they have no current generations of Raymond Chen / that we know of. | |
| ▲ | sterlind a day ago | parent | prev | next [-] | | talent they can hire? large swaths of the company have been in a hiring freeze. they just sent 25% of the company voluntary severance offers. | | |
| ▲ | rsalus a day ago | parent [-] | | they've still been hiring, just on an attritional basis - headcount has remained flat for a few yrs now. it is pretty clear though that they want to operate even leaner. |
| |
| ▲ | lelanthran a day ago | parent | prev | next [-] | | I've been hearing that since 2002, at least. Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option. | | |
| ▲ | disgruntledphd2 a day ago | parent [-] | | > Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option. My suspicion is that lots of the companies are so attached to all the MS stuff (and that's what their staff know), that it would be an incredibly large lift for them to move off it. Like, I'm at a very windows/azure focused shop now (for the first time in my career), and literally everything is already on MS, or moving towards MS. There are some good reasons for that, and some bad reasons, but I suspect that most of the sales process happens at the executive level, rather than the technical level (and Excel runs the world, so why not let MS do everything else?). |
| |
| ▲ | mihaaly a day ago | parent | prev | next [-] | | Jerk-offs can skim the profit of going big yesterday while the consequences affect everyone, but only in the future of yesterday. Who does such things, one might ask? Jerk-offs, off course. And PE too (likely not a disjoint set). | |
| ▲ | kakacik a day ago | parent | prev [-] | | Was there ever a time when this was not true? M$ was always among top hated companies since 90s, arrogance, fuck-U to users all around, absolutely ignoring any standards larger than them, and famous 'Embrace, extend, and extinguish' which was pathetic to watch for any engineer worth 2 cents. Top notch user experience peak was Windows XP and every subsequent release was massively larger, clunkier and slower while requiring way more resources, was spying on the users more and more. Office has similar story, peak of usability was maybe 2007 with ribbon introduction (which still a lot of power users hated with passion since it was far from clean design, more seemingly designed by committee of marketing guys). |
| |
| ▲ | Cthulhu_ a day ago | parent | prev | next [-] | | This is what I don't get about big companies and their software; you'd think every application, every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something, ranging from (in microsoft's case) Paint to a particular configuration panel. I get that this doesn't scale though - a lot of features are just "done" at some point and keeping one or more people whose sole job is to maintain this may lead to them bolting on features just to stay relevant / stop being bored. But big tech can easily afford to keep people like that on. (I mean they keep (kept?) tons of middle management on, who knows what they even do? Not emailing right now that's for sure am I right?) | | |
| ▲ | as1mov a day ago | parent | next [-] | | > every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something, I work at a company with a few thousand employees in the Netherlands, and have been moved around a few teams in the last 7 years (not by choice, the team is nuked and I'm shuffled off elsewhere). Technically, the product is supposed to have a "handover" to another team who can do the maintenance. What happens in practice is the person who worked on it gets saddled with maintaining it forever along with taking up the responsibilities of the newer work that comes along their way. I've accumulated a string of such projects, including one that's used extensively by a few large companies, and the only person maintaining it is drumroll ... me, in whatever time I can spare for it. | |
| ▲ | ctmlr a day ago | parent | prev [-] | | Companies just don't want to compensate people to hang around and maintain a system, they want a consistent output of widgets that can be measured. Somehow our work culture is still stuck in factory worker mentality despite being well into the information revolution now. |
| |
| ▲ | logicchains a day ago | parent | prev [-] | | >the Visual C++ standard library has officially one FTE working on it One guy but it's Stephan T. Lavavej the guy who the "STL" was literally named after. | | |
|
|
| ▲ | EmX_3000 a day ago | parent | prev | next [-] |
| Mythical man month, maybe throwing everybody in the company at it isn't the right thing to do. But they must be in a panic, and their own communications internally may well be hampered, could be MS everything is out for MS as well as everybody else. |
| |
| ▲ | TheOtherHobbes a day ago | parent | next [-] | | Someone's going to have to write the Mythical AI Month after this. If you lose 90% of developers to AI, but everything breaks, how much money have you saved? The underlying moral issue is that the management and money people have contempt for people who don't manage and don't have money. Which includes customers and engineers. It's not just an inhumane system - although it is, that, brutally - but it's also technologically dysfunctional and ultimately unstable. | | | |
| ▲ | flir a day ago | parent | prev [-] | | I've seen the response to an emergency be "get every dev in a big room with some non-technical middle managers to direct them, and have everybody talk at once". My jaw was on the floor. "When in danger, when in doubt, run in circles, scream and shout". |
|
|
| ▲ | nikanj a day ago | parent | prev | next [-] |
| In our field, once you understand what the problem is, the problem is usually solved. Unless it's broken hardware, obviously |
| |
|
| ▲ | ChrisRR a day ago | parent | prev | next [-] |
| At least 1 |
|
| ▲ | InsideOutSanta a day ago | parent | prev | next [-] |
| The ones who actually knew how the system works were too expensive to retain, so now it's just some interns typing frantically into Claude Code. |
| |
| ▲ | digikata a day ago | parent | next [-] | | "Too expensive" as measured by bean counters that don't understand the actual risk profile - or do understand it but have such a short window of strategic concern it leads to this. It would be an interesting analysis to stack up costs to Msoft vs cost of labor cut as this outage evolves. | |
| ▲ | StingyJelly a day ago | parent | prev | next [-] | | Company policy is to use copilot but copilot is down so they are just hitting F5 and hoping | | |
| ▲ | aucisson_masque a day ago | parent | next [-] | | Haha this is ridiculous and yet there is some truth in it. | | |
| ▲ | kakacik a day ago | parent [-] | | It may not be true now but 10 years down the line? This is disaster in the making yet while most people recognize that (when filtering out most fanatical pro-ai folks claiming everywhere how they get 10x or 100x work done) nobody is able to do anything. Sort of legacy technical debt being delivered but of different kind than usual, that train that will hit you since you stand on the rail line yet its moving 5kmh and you see it from 200m, but unable or unwilling to move away (Austin powers scene with steamroller does it better than me [1]). Imagine if say BMW decided they don't want to manufacture their motors, suspensions, drivetrain etc. Outsourcing 100% of that knowledge to external suppliers, for all models, from now on forever. How do you think market would reacted to such move ie by valuation of stocks? [1] https://www.youtube.com/watch?v=y_PrZ-J7D3k | | |
| ▲ | aucisson_masque 16 hours ago | parent | next [-] | | When you outsource everything, you’re not relevant anymore. I agree on that, anyone else can do it just as good as you. But the biggest fear is how is ai going to improve then ? We’ve seen it can’t feed itself on data it produced, the model quality decrease. But if there is no more human to produce ever better quality code because they rely on llm to generate code, we’re going to reach a wall. | |
| ▲ | tonyedgecombe a day ago | parent | prev | next [-] | | Talking of car manufacturers: https://en.wikipedia.org/wiki/Jaguar_Land_Rover_cyberattack Now imagine the consequences if this was the main supplier of insulin. | |
| ▲ | smackeyacky a day ago | parent | prev [-] | | Given how unreliable and poorly engineered modern BMWs are the market might react positively |
|
| |
| ▲ | SillyUsername a day ago | parent | prev [-] | | This should be the top comment, my eyes are watering to this one |
| |
| ▲ | InsideOutSanta a day ago | parent | prev [-] | | Oh shit: https://status.claude.com/incidents/nn11wk485zfn Microsoft can't fix Outlook because they don't have Claude, and Anthropic's employees can't fix Claude because they can no longer log in without Outlook? | | |
|
|
| ▲ | micw a day ago | parent | prev | next [-] |
| > How many Microsoft employees are working on this? all 3! |
| |
| ▲ | digikata a day ago | parent [-] | | They're more effective than ever because with this outage, every manager above them can't email to ask for status every 5 min | | |
| ▲ | pferde a day ago | parent | next [-] | | Alas, Teams are working, or at least the chat and voice call portions of it. So managers are in a calling and chatting frenzy right now, pulling every body else in. | | |
| ▲ | kakacik a day ago | parent [-] | | search in teams doesn't, very convenient for our Windows team to be unreachable for those of us still on WFH |
| |
| ▲ | motionlessveloc a day ago | parent | prev [-] | | Fortunately they can still ask on Teams! |
|
|
|
| ▲ | russfink a day ago | parent | prev | next [-] |
| I wonder if it’s an AI death spiral: “the service is down. Diagnose the failure.” … Hello? |
|
| ▲ | karlkloss a day ago | parent | prev | next [-] |
| How may Microsoft employees are left anyway? |
| |
| ▲ | phoronixrly a day ago | parent | next [-] | | The lights are on but there's no one home | | | |
| ▲ | wolfi1 a day ago | parent | prev [-] | | I guess the old light bulb jokes about MS will not be applying anymore |
|
|
| ▲ | boudin a day ago | parent | prev | next [-] |
| /s Their internal AI is full steam on it. The issue is just that their internal AI infrastructure is down too, so AI just need to fix itself first. Then it will focus on fixing the issue. Oh, wait... |
|
| ▲ | wooger a day ago | parent | prev | next [-] |
| And which made a load of staff redundant in the last month. |
|
| ▲ | surgical_fire a day ago | parent | prev | next [-] |
| This is HN. We should be asking why frontier models couldn't still dolve this! How many parallel agents does MS have working on this? |
|
| ▲ | VariousPrograms a day ago | parent | prev | next [-] |
| 211 muted people on the Teams call, 2 guys trying to figure it out, an incident manager asking for an ETA and status update every so often. |
| |
|
| ▲ | a day ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | DarkCrusader2 a day ago | parent | prev | next [-] |
| I don't understand why they haven't asked copilot to fix it yet, make no mistakes. |
| |
| ▲ | mint5 a day ago | parent | next [-] | | Don’t be silly, they tried that but copilot is/was down! That’s why it’s taken over 12 hours now, no one’s left that knows without copilot’s help. | |
| ▲ | cokeandpepsi a day ago | parent | prev [-] | | [dead] |
|
|
| ▲ | xyst a day ago | parent | prev | next [-] |
| the vibe coding group in charge of this product has decided to delete the production database and backups |
|
| ▲ | krige a day ago | parent | prev [-] |
| Maybe Nadella is still trying to figure out which of the 8000+ sacked employees knew how the system works /s |
| |
| ▲ | sterlind a day ago | parent [-] | | he was such a nice and effective CEO the first few years. he's really changed. | | |
| ▲ | kakacik a day ago | parent | next [-] | | Was he? I struggle to see it even in hindsight, and definitely his moves were bunch of FU to common users. Maybe in some specific area you care about, but not their core products like windows and office | |
| ▲ | cyberax a day ago | parent | prev | next [-] | | Not really. He killed the QA department immediately and consciously put MS onto the "slide into mediocrity" path. | | |
| ▲ | motionlessveloc a day ago | parent [-] | | Google killed its test engineers (or, rather, forcibly merged them into SWE) at around the same time without measurably denting uptime. | | |
| |
| ▲ | pjmlp a day ago | parent | prev | next [-] | | Even with some of the Microsoft <3 Open Source, I do miss the Steve Balmer era in regards to Windows development experience. | |
| ▲ | bell-cot a day ago | parent | prev | next [-] | | Executive compensation corrupts, chief executive compensation corrupts absolutely. | |
| ▲ | Markoff a day ago | parent | prev [-] | | yeah very effective, he fired 90% of our Windows Store team (reviewing the apps) in the first year to replace people with automation, I survived up to the last round of firing after which they kept ~20 employees out of original ~200 |
|
|
