Remix.run Logo
▲ bayindirh a day ago

Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.

So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.

▲lokar a day ago | parent | next [-]

Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.

This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.

▲bayindirh a day ago | parent [-]

Sounds reasonable.

I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?

▲lokar a day ago | parent | next [-]

IP is often hard on the proxy/load balancer, you get a really large list, which becomes expensive.

▲bayindirh a day ago | parent [-]

This is also what I thought, so why I asked, actually.

▲toast0 a day ago | parent | prev [-]

Selectively allowing IPs is a reasonable way to do it. If you think about it, it's terrible, but it will get you going, and it's pretty easy do something like start with allow nothing, then allow all ips under 2.0.0.0, then 3.0.0.0, etc.

Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.

You could do similar in IPv6, start allowing IPs under 2001::, and bump it up little by little... probably each increment would be the 2nd byte, but you do need to increase the first byte too ... or you could be a terrible person and drop ipv6 while you modulate ipv4 and only let ipv6 in when ipv4 is at 100%.

Works better the earliest you can apply it... if your whole stack passes through requests and the origin is looking at X-Forwarded-For (or similar) headers to drop requests, your whole stack worked on it for nothing... Which could be ok, but your frontends might be much happier if they just dropped SYNs from IPs when the request will fail.

▲DenisM 7 hours ago | parent [-]

I’m wondering if you’re preparing for cold start ahead of time you could create a better cold start IP range map that’s distributed more evenly for more predictable turn up, and also with better affinity so you can you report progress in a way people can understand ( eg state by state). It would be a pain to load and activate the map into edge routers in a hurry, but maybe be necessary. Some superscalers are deep into writing their own networking firmware, this could go into design requirements for text revision.

Another idea is to have different clients use different ports (enterprise customers on one port, SMB on another) and turn up in the order of SLA.

▲nijave a day ago | parent | prev [-]

They could drop traffic at L3 before it even hits their servers.

▲lokar 9 hours ago | parent [-]

That's what we are talking about. But you need a way to manage that L3 traffic by client.