Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.

- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.

I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.

> This is 100% fault of the hotel, not of Microsoft.

yeah, these companies need to be told to stop relying on cloud services for everything.

its wild to me how many critical things that absolutely do not need the internet are relying on the internet.

much blame to go around here:

- whoever designed the stupid alert system.

- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”

- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”

- accountant types and their “we can save pennie’s if we cut all of these corners.” and the stupid corners they choose to cut.

- etc…

on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.

This seems like the kind of thing which should have an audible alarm mounted on the wall in the pool room.

Exactly. Email is also a ‘zero guarantee’ service, and should never be relied upon for life or death solutions.

It's the fault of the company that designed the automatic alarm. Not the hotel. But it is accurate to say that the cause of the issue was the outage.

This gives me an idea for installation art:

Title: 5 nines of uptime

A grand piano is suspended from the ceiling with an electromagnet.

The electromagnet works as long as there is power.

No special battery backups, anything.

Guests are invited to stand underneath it, after signing a waiver.

That’s it. That’s the art.

Seriously, what's next -- hotels which decide that smoke detectors ought to e-mail the front desk rather than blaring a loud alarm?

I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?

Seems like if the system can send an email it should be able to, like, illuminate a red light somewhere or something, if what it’s doing can create an actual safety risk.

The law is smarter than this, thankfully.

Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.

And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.

Came here to say this. A push notification to an app or Slack/Discord. Or even a light.

I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.

hindsight is 20/20.

I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.

Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.

In short, all parties suck here (MS and hotel)

I don’t understand why someone would design a pool system that can harm swimmers with chlorine and the only safety is an automated email notification, and then blame the email provider and not their own system design for it.

There's no details in the article so you didn't miss anything. TLDR

> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.

> I work in Azure

I’m so sorry.

excuse my language but holy shit thats fucking stupid. email? for chlorine gas alerts?

Why?

Like, what leads a teenager to do such things?

I don't think "chlorine gas" but rather chloramine. In pools when chlorine binds to contaminates, it forms chloramine, which is a gas that sits on the surface of the pool.

No, there are many Linux users there, and some small high profile government cases of trying to move of office, but like everywhere else it is hugely dominated by Windows, especially this type of smaller business area, hotels, OT systems etc.

Seems like all of Europe is more tied to Microsoft than the US, oddly.

We have news articles of emphasis on Linux. That’s enough.

I didn't know Torvalds ran his own email service

Email as the sole notification source for a safety critical situation is criminal, in the literal “the responsible engineer or coder masquerading as an engineer should be in jail” sense of the term.

We all know that services can fail, and consequently will fail at some moment. It is therefore primarily the responsibility of the designers of this system.

They should have closed the pool.

Strange thing to drag our entire country in just because one hotel had a very stupidly designed alert system.

This is an automated system almost certainly. It monitors the chemistry and sends alerts if things fall out of range.

Apparently the alarm system sent its alarms via email.

"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."

We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.

It’s a neuland for them after all.