We learned the hard way that there's a very non-obvious inflection point in large distributed systems, where the your design needs to change from accommodating load resiliently via queueing, auto scaling, and such, to a posture of actively scaling back certain features and even shedding load when in a defensive situation.

It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.