It's very common, but typically teams test things by having a staging instance which also has staging versions of all dependencies.

They then deliberately loadtest it to failure and check it can be bought back under control.

▲

voidfunc 16 hours ago | parent [-]

The staging pattern doesn't scale for systems this large and complicated. Everything is constantly in flux and changing. Ive worked on systems this large and we had something we called staging but it was a joke.

Gradual rollout, pilot regions and monitoring are usually how you avoid broken stuff becoming global incidents. Will be interesting to read what process failed here next week.

	▲	londons_explore 12 hours ago \| parent [-]
		This type of loadtest I normally only perform ~annually in the companies I have done it. It's normally a ~2 week-long project for an intern to identify failure modes of any client side apps (ie. Retry loops etc), capture a sample of production API requests, create some fake data store either on staging or an instance setup specially for the purpose, adjust the request sample to hit that fake data (reads, writes etc), then replay the sample faster and faster till staging fails due to overload, and practice a recovery procedure with the load still applied. If it goes well, redo it with different types of overload - especially large percentages of failure responses from various dependencies, or 1000ms delays added to dependency responses which tends to be one of the trickier things to recover from. Typical outcomes from the work are patches to make the service more robust, fallbacks to not use dependencies or disable features when overload happens, on-call procedures for what to do or not do in certain common cases etc.