Then your load balancer becomes the single point of failure.

Then add a load balancer in front of your load balancer, duh. /s

BGP Anycast will let you dynamically route traffic into multiple front-end load balancers - this is how GSLB is usually done.

Needs an ASN and a decent chunk of PI address space, though, so not exactly something a random startup will ever be likely to play with.