What happens when the load balancer fails?

HA has to be all the way through, in which case you might not need a load balancer because each client already connects to a separate server. If you do, then you can have one load balancer per client machine.