I once accidentally rebooted the reverse proxy for all our production traffic. We got some very quiet two minutes while it came back up.

After that we installed molly-guard with a check for the number of active connections. Made it painless to reboot standby proxies and difficult to reboot active ones.

(We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)

I like telling junior hires about this incident because it teaches them that (a) anyone can make mistakes, (b) even serious mistakes usually aren't that dangerous, (c) you can learn a lot from mistakes with the right mindset, (d) we cannot prevent mistakes but with the right system design we can reduce their consequences.