Give that engineer a raise and an official title of "chaos monkey".

On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.

Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.

Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.

A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.

---

¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.