I’ve seen this up close twice and I’m surprised it’s only twice. Between March and September one year, 6 people on one team had to get new hard drives in their thinkpads and rebuild their systems. All from the same PO but doled out over the course of a project rampup. That was the first project where the onboarding docs were really really good, since we got a lot of practice in a short period of time.

Long before that, the first raid array anyone set up for my (teams’) usage, arrived from Sun with 2 dead drives out of 10. They RMA’d us 2 more drives and one of those was also DOA. That was a couple years after Sun stopped burning in hardware for cost savings, which maybe wasn’t that much of a savings all things considered.

I got burnt by this bug on freakin' Christmas Eve 2020 ( https://forum.hddguru.com/viewtopic.php?f=10&t=40766 ). There was some data loss and a lot of lessons learned.

Many years ago (13?), I was around when Amazon moved SABLE from RAM to SSDs. A whole rack came from a single batch, and something like 128 disks went out at once.

I was an intern but everyone seemed very stressed.

I love that "Ask HN: What'd you do while HN was down?" was a thing

Man I hit something like that once, a SSD had a firmware bug where it would stop working at an exact number of hours.

Yeah looks like they're back on M5.

dang saying it's temporary: https://news.ycombinator.com/item?id=32031136

    $ dig news.ycombinator.com

    ; <<>> DiG 9.10.6 <<>> news.ycombinator.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54819
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;news.ycombinator.com.  IN A

    ;; ANSWER SECTION:
    news.ycombinator.com. 1 IN A 209.216.230.207

    ;; Query time: 79 msec
    ;; SERVER: 100.100.100.100#53(100.100.100.100)
    ;; WHEN: Wed Oct 29 13:59:29 EDT 2025
    ;; MSG SIZE  rcvd: 65