| ▲ | AbbeFaria 21 hours ago |
| Ex-MSFT here, sharing my outdated POV based on working on a product that was using Microsoft Exchange and therefore was hooked onto the email delivery pipeline. Note that the infra for M365, Azure etc is different. They would be running on servers hosted in MSFT DCs but have completely different CI/CD pipelines. M365 services have their own deployment pipelines and it takes on the order of weeks to deploy your changes to production. Considering the fact that Outlook is down, clearly email doesn’t seem to be working. So email uses something called Microsoft Exchange Server, there used to be an on-premises version of this but MS stopped supporting it to run the cloud version hereby called M365. Email is one big “macro-service”, if you’re running Windows Server, look at the process called “EdgeTransport.exe” this is responsible for sending/receiving email on Windows Servers. There are probably multiple 100s of services supported by 100s of teams that are run while email is routed from sender to receiver via MSFT Exchange servers. My recollection is that even if one of these services goes down or introduces latency in the pipeline, it caused a major incident. When I was working in M365 Security in 2024, this Exchange Server pipeline was called “legacy” because it’s one big .NET Framework service. I don’t know if they upgraded to .NET core but developing on this was as HN comments say, very janky. The reason Teams doesn’t seem to be down is that Teams services are completely different and are written using .NET Core and have their own separate release pipeline. It was much easier to interact with Teams services. I actually worked closely with the team that was responsible for ingesting Teams messages to the customer’s data store. Their code base was much more modern and easy to understand. Exchange Server not so much. I don’t know if what I said is accurate but know that Exchange server code was like a fossil that didn’t change as much, over the ~5 years I worked on it from 2019-2024. Story time over. |
|
| ▲ | EvanAnderson 14 hours ago | parent | next [-] |
| > So email uses something called Microsoft Exchange Server, there used to be an on-premises version of this but MS stopped supporting it ... On-premises Exchange is still supported and licensed. It's a subscription instead of a perpetual license. > I don’t know if what I said is accurate but know that Exchange server code was like a fossil that didn’t change as much, over the ~5 years I worked on it from 2019-2024. Exchange is a mature product (albeit not without its flaws). Its era of change was between its 2007 and 2013 versions (inclusive) when it was made more modular and many parts were ported to the .NET runtime. The 2016, 2019, and now Subscription Edition codebases were more of an evolution and has much less churn. |
| |
| ▲ | bigfatkitten 7 hours ago | parent | next [-] | | Microsoft will still charge you money for Exchange SE, and occasionally release security and other bug fixes, but “supported” is an extremely generous description of the state of the product. | | |
| ▲ | EvanAnderson 6 hours ago | parent [-] | | Realistically, though, when it comes to support that's every MSFT product unless your org is shelling-out support dollars in advance. | | |
| ▲ | bigfatkitten 6 hours ago | parent [-] | | The difference with their other products is that they generally want you to adopt it, and they actually do feature development, neither of which are true for Exchange SE. | | |
| ▲ | EvanAnderson 5 hours ago | parent [-] | | My take is they want anything on-prem to go away. Exchange, in particular, is sort of "feature complete" to me. I'm happy w/ what it does and how it does it for the most part. |
|
|
| |
| ▲ | j45 11 hours ago | parent | prev [-] | | Exchange should be well protected behind a firewall or 10 after all these years. What this leaves is a comparison of whether on-premise exchange servers have been compromised or down compared to the cloud. |
|
|
| ▲ | tallanvor 18 hours ago | parent | prev | next [-] |
| This hasn't been the case for a long time. Unfortunately there are micro services that can still bring down the system if there's a bad enough bug. That's not supposed to be able to happen as updates roll out over a period of time. Unfortunately you can still get bit if another update creates a bug that takes out instances faster than they can be recreated. |
| |
| ▲ | j45 11 hours ago | parent [-] | | I would hope that enough hardening has happened over the last 20 or 30 years. |
|
|
| ▲ | 13 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | igleria 19 hours ago | parent | prev | next [-] |
| oof, here in Ireland outlook still down this morning. teams seems to be there tho. |
| |
|
| ▲ | senectus1 20 hours ago | parent | prev | next [-] |
| we seem to be unable to start new chat sessions with people we haven't chatted with internally before. so team is at least partially down. |
| |
| ▲ | bgarbiak 19 hours ago | parent | next [-] | | From the Health Status page: "Microsoft Teams: Most of the core functionality including joining calls, meetings, group chats, and channel messages are not impacted.
Scenarios that depend on Exchange Online, including Calendar related failures with scheduling, invites, viewing, and meeting edits. Creation of Live or Virtual events fail. Failures with Copilot responses for outlook, calendar, or M365 files, read history, or adding to new chats. Navigating to App chats, adding new contacts or syncing contacts in settings fail. Standard channels are working fine but users may experience the inability to create or delete private or shared channels." | |
| ▲ | Aloha 20 hours ago | parent | prev | next [-] | | I suspect that the search is broken because it relies on the exchange bits to find users. | | | |
| ▲ | apexalpha 19 hours ago | parent | prev [-] | | Teams relies on Exchange API to autocomplete your search for colleagues, if you know someone's full email you should be able to start it that way. |
|
|
| ▲ | 13 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | jfsebastian 17 hours ago | parent | prev | next [-] |
| What is driving me nuts is that we still heavily rely on email. Even to the point were we are unable to get notifications other than that. I work at a fortune 500 company and they got me, 3 hours after I started working , the basic info about the obvious the internet is already screaming at you. Since we are responsible for a large web application I tried to reach out to people through VIVA (yes, we only make use of that) and we managed only to reach about 2-3% of the user base. Information-wise it's like a blackout. For living in 2026 this is completely unacceptable. |
| |
| ▲ | chrisandchris 14 hours ago | parent | next [-] | | > What is driving me nuts is that we still heavily rely on email. That itself wouldn't be a problem - email is very decentralised. The problem is that we thought that putting like 2 companies in charge of most email communication is a good idea. | | |
| ▲ | xp84 12 hours ago | parent | next [-] | | Yeah I don’t think anyone wanted that, we just ended up this way because it makes spam control more convenient. Without spam, email is incredibly simple to self-host. I suspect it may have ended up one of the most common services for companies to self-host. | |
| ▲ | cekanoni 13 hours ago | parent | prev [-] | | amin to that! |
| |
| ▲ | franga2000 6 hours ago | parent | prev | next [-] | | Meanwhile we were very happy that we mainly rely on email. We were still able to send out urgent messages (like "sorry if we don't reply, Microsoft is down, you can reach us by [phone] or [backup email]") via SMTP and if we really needed to receive some email, we'd be able to set up a secondary MX record to a holding server in about an hour. If we were mainly using Teams or whatever other "better than email" service and that had an outage, we'd actually be in the dark! As the resident complainer about Microsoft lock-in, I was honestly quite surprised by how well we could work around this. Don't get me wrong, if it was Entra that fell over, we'd be thoroughly fucked, but Exchange wasn't all that bad. | |
| ▲ | xp84 12 hours ago | parent | prev | next [-] | | What’s worse is how many things force an email second factor now. No choice to make a TOTP or to use a passkey (which, with biometrics securing it, is two friggin factors at once, and should be all we need for anything that isn’t nuclear secrets!!!! Sorry this just kills me with every SaaS product I have to “two-factor” to daily). Your email is down? Cool now you can’t log into anything. | |
| ▲ | jasonvorhe 15 hours ago | parent | prev [-] | | Email has been quite robust over the past~ 30 years. Exchange is just email by Microsoft so of course it'll be more brittle. |
|
|
| ▲ | icepush 21 hours ago | parent | prev | next [-] |
| "So email uses something called Microsoft Exchange Server" Is this what it feels like to be old? |
| |
| ▲ | Frost1x 15 hours ago | parent | next [-] | | It’s interesting how relative this is, reading by the comments and stories around them. A previous coworker started computing in the punch card era and would tell stories of dropping your stack of cards and having to reorder them, using the printouts to debug, and so on. And helping develop what’s now known as fork() for processes (although it predated fork and he had his own name for it), and helping develop a commercial RDBMS before they existed all in a non tech industry career. So when I read about Exchange Server to me it still feels quite modern. | | |
| ▲ | dieselgate 7 hours ago | parent [-] | | My father has told some stories about waiting in line to run punch card programs and people dropping their stack and becoming flustered about it. They're funny to hear and mentioning it because this is the first I've ever heard it mentioned elsewhere. |
| |
| ▲ | cowlevel 19 hours ago | parent | prev | next [-] | | Don't frame it as a negative. This is what it feels like to know things that other people don't know. | | | |
| ▲ | AbbeFaria 21 hours ago | parent | prev | next [-] | | To test our code changes, we would check out a VM that was running Microsoft Windows Server edition which was a beefy VM that was running Exchange. It was a carbon copy of the server running in production. The Exchange code base frankly was a behemoth. No one had a full understanding of how it worked E2E. I am just sharing how I understood it. I was only a lowly L63 btw. | | |
| ▲ | wting 20 hours ago | parent [-] | | Explaining Exchange to us older folks is like explaining an iPhone to younger generations. :p As an aside, I worked on Lotus Domino—IBM's competitor to MS Exchange—before it was sold it off in 2018. | | |
| ▲ | reddalo 19 hours ago | parent | next [-] | | >Lotus Domino Oh, the memories. That login screen with the changing hieroglyphics [1] is forever burned into my mind. [1] https://web.archive.org/web/20080220070921/http://lotusnotes... | | |
| ▲ | nycdotnet 7 hours ago | parent | next [-] | | email is red until it’s read, and then it’s not red anymore. | |
| ▲ | cowlevel 19 hours ago | parent | prev [-] | | > The hierglyphics on the left of the dialog box are supposed to distract someone who is peering over your shoulder trying to learn your password as you type. lmao... but is it actually random or is it a hash of your password to make sure you typed it correctly? | | |
| ▲ | j16sdiz 18 hours ago | parent [-] | | iirc, its a hash with server secret. The idea was you can tell if the dialog is the real one, not just some look-the-same dialog from random malware |
|
| |
| ▲ | rvba 18 hours ago | parent | prev | next [-] | | why did Lotus desktop download emails only every 15 (?) minutes? I remember people phoning to say "I"ve sent you an email, press the manual refresh button to get it faster". up to this day I ask if companies use Lotus and those who remember just shake their heads | | |
| ▲ | pasc1878 17 hours ago | parent | next [-] | | All email was like this at one time. In the 90s we were told to expect emails to take a few hours. If you want immediate response use IRC or actually phone someone. | | |
| ▲ | bradleyjg 15 hours ago | parent | next [-] | | If was like this because it was designed before networks were expected to be fully connected at any one time. Mail would hop its way towards its destination as intermediate nodes dialed in over modem to upstreams. | | |
| ▲ | dd8601fn 13 hours ago | parent [-] | | It was like that because each check/send process was pop3 call that looked more like processing a queue of work for offline clients. Now it’s more like being subscribed and listening for events. | | |
| |
| ▲ | icedchai 13 hours ago | parent | prev [-] | | I had email at my house, in the early 90's, over a UUCP feed. It polled every 4 hours. |
| |
| ▲ | xp84 11 hours ago | parent | prev | next [-] | | Think about it in context. Polling constantly would consume a non-trivial amount of both bandwidth and server resources. 15 second intervals would require 60 times as many very expensive servers to handle a certain number of users as 15 minutes would. And in the 1990s, people were just adopting email, and their existing context was mail that arrived daily. Other than IRC, nobody had ever experienced any instant instantaneous form of communication other than phone calls, so nobody really felt like they were missing anything if they didn’t get notified about an email for 15 minutes. Today by contrast, I feel like 3/4 of the time I go into my email client it’s to get some login code sent by some stupid website, so from this perspective 15 minutes sounds like absolute torture. | |
| ▲ | SoftTalker 11 hours ago | parent | prev | next [-] | | I still manually check for mail. I don't want distracting notifications popping up at random. | |
| ▲ | j45 11 hours ago | parent | prev [-] | | This seemed a little hard to believe but it sounded like it was more of a replication where it was downloading things into an offline mode where you then write your emails and send them back again period. Other email clients might do this as well if you're running it on your desktop. |
| |
| ▲ | Reason077 10 hours ago | parent | prev [-] | | “Domino” just seems like a really bad name for a server/cloud product. The memes! The analogies! |
|
| |
| ▲ | j45 11 hours ago | parent | prev [-] | | I doubt it has something to do with being old, running an email server of any kind, let alone knowing how email works, let alone using email, have been all various abstractions to help make users into consumers who pay more than people who know how to build and run this infrastructure. |
|
|
| ▲ | hobofan 20 hours ago | parent | prev | next [-] |
| > Microsoft Exchange Server, there used to be an on-premises version of this but MS stopped supporting it After many enterprise customers refused to migrate Exchange Server SE exists since 2025 and will be supported until at least 2035. Though given how the quality of their updates are going one could still say that it's "not supported", but at least they are charging money for it! |
| |
|
| ▲ | cmckn 21 hours ago | parent | prev | next [-] |
| If that’s all you’ve got, I’d probably scrub this and keep your NDA intact. |
| |
| ▲ | AbbeFaria 21 hours ago | parent [-] | | This is common knowledge. Try googling EdgeTransport.exe and you should see the outdated documentation for Exchange Server on-Prem. |
|
|
| ▲ | Betelbuddy 19 hours ago | parent | prev | next [-] |
| Ok but why would all infra would go down everywhere, at the same time? Is it all a big server? Are there any Architects left or is it all Golf Players? |
| |
| ▲ | NetMageSCW 12 hours ago | parent | next [-] | | It’s a distributed system with thousands of servers with automatic update distribution and config change distribution. Bad changes can propagate across the entire system though Microsoft status updates say they didn’t see any change that seemed relevant. It could also be a bug across all the uniform code that somehow triggered. For example Cisco BE servers once had a firmware bug that after 32767 hours the fans would all run at 100% speed. | |
| ▲ | jasonvorhe 15 hours ago | parent | prev [-] | | Microsoft product reliability has been a running joke since the company started launching products. Why would anything change just because they pivoted to managed services instead of shipping install media? |
|
|
| ▲ | hyperman1 21 hours ago | parent | prev | next [-] |
| If it takes weeks to get some code into production, I understand how AI can enormously speed up things. Yes, AI can be fantastic but if people claim 100x increase, something else is majorly fubar. |
| |
| ▲ | user43928 16 hours ago | parent [-] | | I don't know what OP means. The deployment pipeline takes weeks or the release process? The latter is probably the case for most enterprises. Getting releases deployed into the testing environments, much less production, requires coordination, testing, reports and sign-offs. Of course this can normally be expedited or bypassed for incident management and critical fixes. |
|
|
| ▲ | glitchbot 13 hours ago | parent | prev | next [-] |
| I see Walker and Texas ranger running around screaming ANARCHY? |
| |
| ▲ | reaperducer 11 hours ago | parent [-] | | All I see today is middle managers using an e-mail outage as an excuse for not getting things done yesterday. Pick. Up. The. God. Damned. Phone. And do your job. | | |
| ▲ | nomadygnt 10 hours ago | parent [-] | | I'm not a middle manager and in general I agree with you. But I tried to log in and it sent a code to my email. Whelp... I'll do what I can I guess. |
|
|
|
| ▲ | jjav 18 hours ago | parent | prev [-] |
| All the MS memes today made me realize people still use Outlook? I must live in a bubble... the last time I heard of anyone using Outlook was when Scott McNealy was calling it LookOut! in the 90s. In a ~35 year tech career I have never been exposed to, nor used Windows nor Outlook and have never met anyone who does. Who are these companies that still use it? |
| |
| ▲ | oytis 18 hours ago | parent | next [-] | | It's not about outlook the mailing app. They called their SaaS e-mail Outlook, you can use it without using Windows, in the same way as Gmail. Pretty popular in the corporate world | |
| ▲ | gpvos 17 hours ago | parent | prev | next [-] | | A large majority of companies I've worked for over the last 30+ years used Outlook and Exchange Server, or now M$365, for email, even in those cases when my actual work was on Linux. The exception were those that used Gmail... | |
| ▲ | w4der 18 hours ago | parent | prev | next [-] | | Many, many big companies do, and some universities also moved to MS365 when Google downgraded their free campus agreement, because they were already paying for MS365 for Office and that came with cloud storage and outlook. | |
| ▲ | lxgr 16 hours ago | parent | prev | next [-] | | > I must live in a bubble... the last time I heard of anyone using Outlook was when Scott McNealy was calling it LookOut! in the 90s. [...] In a ~35 year tech career I have never been exposed to, nor used Windows nor Outlook and have never met anyone who does. Yes, then you probably at least work in a bubble. In my experience, it's more common than GSuite in most companies outside of startups and big tech and is ubiquitous at schools, among many, many other entities. | | |
| ▲ | dmd 15 hours ago | parent | next [-] | | Of the ~40 companies I work with on a daily basis, 40 use Outlook. | |
| ▲ | swasheck 14 hours ago | parent | prev [-] | | I’ve worked at 2 companies who used Google for email and calendaring but still relied on O365 (one of the many naming incarnations) for Word, Excel, and PowerPoint. Microsoft really do have quite a mindshare foothold in the office productivity market. | | |
| ▲ | wcfields 10 hours ago | parent [-] | | Last big corp I worked for had a half-half setup of Outlook for email (and you were reminded that messages are retained for 7 years) and used Zoom+Slack instead of Teams (it was disabled, so not even an option), but we did use Sharepoint. |
|
| |
| ▲ | phatfish 18 hours ago | parent | prev | next [-] | | Most of the Fortune 500? | | |
| ▲ | infecto 17 hours ago | parent [-] | | Probably all of the Fortune 500 except some of the tech companies. | | |
| |
| ▲ | infecto 17 hours ago | parent | prev | next [-] | | Sounds like you live in the Silicon Valley bubble. I don’t have a stat but I would wager that most businesses (especially larger ones) run on outlook/exchange. You have probably mostly worked for tech companies where business employees were not the primary workforce. When you think about it most other business generally have everyone needing a word, excel or PowerPoint. It bundles cleanly with the whole package. | |
| ▲ | martin- 17 hours ago | parent | prev | next [-] | | Windows is still by far the most popular OS for employees at big companies. Using Microsoft's SharePoint/Outlook/etc is also extremely common. I'm assuming your "surprise" isn't genuine. | |
| ▲ | Iulioh 18 hours ago | parent | prev | next [-] | | Every non tech company uses outlook, basically. | |
| ▲ | LegitShady 17 hours ago | parent | prev | next [-] | | It's interesting what kind of bubbles people live in. | |
| ▲ | Guy-ute 13 hours ago | parent | prev | next [-] | | You are absolutely living in a bubble. The vast vast majority of US companies are using both Windows and Outlook. | |
| ▲ | 7bit 17 hours ago | parent | prev | next [-] | | [dead] | |
| ▲ | lokar 13 hours ago | parent | prev [-] | | I’m with you. I took a job in finance at one point, reporting to the CTO. On the first day they showed me my desk, a windows machine. I had not used it since high school, and not seriously then. It just never occurred to me that you would have engineers use windows (all the trading systems were Linux). The CTO had to pull a bunch of strings to let me run Linux. |
|