I’ve seen this up close twice and I’m surprised it’s only twice. Between March and September one year, 6 people on one team had to get new hard drives in their thinkpads and rebuild their systems. All from the same PO but doled out over the course of a project rampup. That was the first project where the onboarding docs were really really good, since we got a lot of practice in a short period of time.

Long before that, the first raid array anyone set up for my (teams’) usage, arrived from Sun with 2 dead drives out of 10. They RMA’d us 2 more drives and one of those was also DOA. That was a couple years after Sun stopped burning in hardware for cost savings, which maybe wasn’t that much of a savings all things considered.

I got burnt by this bug on freakin' Christmas Eve 2020 ( https://forum.hddguru.com/viewtopic.php?f=10&t=40766 ). There was some data loss and a lot of lessons learned.

Many years ago (13?), I was around when Amazon moved SABLE from RAM to SSDs. A whole rack came from a single batch, and something like 128 disks went out at once.

I was an intern but everyone seemed very stressed.

I love that "Ask HN: What'd you do while HN was down?" was a thing

Man I hit something like that once, a SSD had a firmware bug where it would stop working at an exact number of hours.

Yeah looks like they're back on M5.

dang saying it's temporary: https://news.ycombinator.com/item?id=32031136

    $ dig news.ycombinator.com

    ; <<>> DiG 9.10.6 <<>> news.ycombinator.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54819
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;news.ycombinator.com.  IN A

    ;; ANSWER SECTION:
    news.ycombinator.com. 1 IN A 209.216.230.207

    ;; Query time: 79 msec
    ;; SERVER: 100.100.100.100#53(100.100.100.100)
    ;; WHEN: Wed Oct 29 13:59:29 EDT 2025
    ;; MSG SIZE  rcvd: 65

Why? Starbucks is not providing a critical service. Spending less money and resources and just accepting the risk that occasionally you won't be able to sell coffee for a few hours is a completely valid decision from both management and engineering pov.

you wouldn't believe some of the crap enterprise bigco mgmt put in place for disaster recovery.

they think that they are 'eliminating a single point of failure', but in reality, they end up adding multiple, complicated points of mostly failure.

It could be local-first. I don't mind the cross-device sync being done centrally, of course, but the app specifically asks for access to Home and Local Network. I wonder if Home Assistant could deal with blackouts…