And you never really get an opportunity to thoroughly test these failure & recovery modes until something catastrophic happens, at which point capturing detailed metrics and assessing the right sequence to perform the actions are both challenging at best.

Yep. There is simply no load testing you can do when you reach certain levels of traffic. Nothing can generate the type of load you get when you have millions of end users making requests from thousands of different networks, on different equipment and network configurations.