"Everything went down, trying to bring back a subset of things gets killed by thundering herd" is very common.

Arbortheus 20 hours ago | parent | next [-]

This is so true. Not making excuses for Microsoft, but OP’s at this very hard. What would have been a “reboot the server” at small scales becomes a concerted operational nightmare at larger scales.

▲

gordonfish 14 hours ago | parent | prev | next [-]

Why so? I'd have thought that in situations like this, one would limit inbound connections, either to certain IPs, or through rate limiting controls, all of which should be controllable at the router / network infrastructure level.

Even more so when dealing with a potential incursion (like servers compromised with botnet software that goes back to work not long after a reboot and slams the server and/or network.)

	▲	adamsb6 11 hours ago \| parent [-]
		Some form of demand control is generally how you work around these problems, but maybe those parts are falling over too. For really big services it's also hard to exercise these processes on anything approaching actual prod scale. Ideally you can chop the architecture into pieces where you can simulate region failure and ensure everything works as planned, and fix whatever you find that breaks with every simulation. But maybe some bits of the architecture aren't amenable to being regionalized.

▲

londons_explore 17 hours ago | parent | prev | next [-]

It's very common, but typically teams test things by having a staging instance which also has staging versions of all dependencies.

They then deliberately loadtest it to failure and check it can be bought back under control.

▲

voidfunc 16 hours ago | parent [-]

The staging pattern doesn't scale for systems this large and complicated. Everything is constantly in flux and changing. Ive worked on systems this large and we had something we called staging but it was a joke.

Gradual rollout, pilot regions and monitoring are usually how you avoid broken stuff becoming global incidents. Will be interesting to read what process failed here next week.

	▲	londons_explore 12 hours ago \| parent [-]
		This type of loadtest I normally only perform ~annually in the companies I have done it. It's normally a ~2 week-long project for an intern to identify failure modes of any client side apps (ie. Retry loops etc), capture a sample of production API requests, create some fake data store either on staging or an instance setup specially for the purpose, adjust the request sample to hit that fake data (reads, writes etc), then replay the sample faster and faster till staging fails due to overload, and practice a recovery procedure with the load still applied. If it goes well, redo it with different types of overload - especially large percentages of failure responses from various dependencies, or 1000ms delays added to dependency responses which tends to be one of the trickier things to recover from. Typical outcomes from the work are patches to make the service more robust, fallbacks to not use dependencies or disable features when overload happens, on-call procedures for what to do or not do in certain common cases etc.

▲

dboreham a day ago | parent | prev [-]