Remix.run Logo
▲ ZiiS 9 hours ago

- we added a bug (yes, everyone does).

- our testing didn't cover it (again, this is expected occasionally).

- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).

- it took 15 hours to deploy the fix (not good, but at this scale maybe).

Feels like they are completely missing the point of where the problem is and how to fix it?

▲topaz0 an hour ago | parent | next [-]

Antitrust

▲VirusNewbie an hour ago | parent | prev | next [-]

>- it took 15 hours to deploy the fix (not good, but at this scale maybe).

I've definitely been in similar scale war rooms and 15 hours is not acceptable.

▲lokar 9 hours ago | parent | prev [-]

In terms of cascading failure and inability to rollback or fix quickly, MS has always been terrible.

▲bayindirh 9 hours ago | parent [-]

Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.

So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.

▲lokar 9 hours ago | parent [-]

Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.

This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.

▲bayindirh 9 hours ago | parent [-]

Sounds reasonable.

I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?

▲toast0 9 minutes ago | parent | next [-]

Selectively allowing IPs is a reasonable way to do it. If you think about it, it's terrible, but it will get you going, and it's pretty easy do something like start with allow nothing, then allow all ips under 2.0.0.0, then 3.0.0.0, etc.

Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.

You could do similar in IPv6, start allowing IPs under 2001::, and bump it up little by little... probably each increment would be the 2nd byte, but you do need to increase the first byte too ... or you could be a terrible person and drop ipv6 while you modulate ipv4 and only let ipv6 in when ipv4 is at 100%.

▲lokar 9 hours ago | parent | prev [-]

IP is often hard on the proxy/load balancer, you get a really large list, which becomes expensive.

▲bayindirh 8 hours ago | parent [-]

This is also what I thought, so why I asked, actually.