To me this makes no sense. Nothing in web development changes because of long running requests, there are plenty of solutions for this. The most easy one is to just listen long enough on a http request for the answer. The routing problem can be mitigated with session pinning. Http2 and 3 have solutions for streaming data, websockets can be used, and pub/sub also. Heck, we could push the LLM response in a k2v system/redis and read it from there. "State is in the DB" is running strong and will be for decades to come.

The industry decided a long time ago that sticky sessions was a terrible idea. They only half-solve the problem, while suffering from session loss on server loss and imbalanced load over time.