Interesting theory. So how does ChatGPT begin responding instantly, as soon as I send the message? Shouldn't it need to wait for the batch to fill? Or do they have so much traffic that this happens in a few ms?

(I think they might also be filling the message onto a GPU while you're typing over a websocket or something, but I'm not sure.)