> What's conspicuously missing is the plot of performance when you do have a well tuned queue in front of the service.

As in between the service and the load balancer? There's already an infinite queue in the load balancer. You can try that out on https://stability-sim.systems/ to see the effect, but the short version is that (in this model) it makes things worse.

If you're saying that the queue in the load balancer should be limited in size to reduce tail latency, then I agree.

▲

megamalloc 7 hours ago | parent [-]

No, I mean when you have a queue broker that the backends can pull work from when they become idle, rather than relying on load balancing which will send work to backends while they're still busy.

▲

wmf 5 hours ago | parent [-]

This scenario already works that way. The very first sentence says "servers, each of which can only handle a single concurrent request, and has no internal queuing". This implies that the load balancer waits for a server to finish a request then immediately sends the next one.

	▲	megamalloc 4 hours ago \| parent [-]
		I don't believe it does. As I understand it, the load balancer has a queue in which it can buffer infinite requests, but it drains that queue by pushing work to the backend servers in what's probably a round-robin fashion. So there is secondary queueing at each server. Even the "least connections" strategies available through some load balancers do not usually behave as you might expect (by always sending the next request to a server that's idle). Pull-based load balancing via a queue has its own downsides but the big upside is to make latency essentially a constant low overhead regardless of the number of servers in the typical case.