But surely it does? It is TCP, even with multiplexing

I thought that was the whole motivation for building HTTP/3 on QUIC (UDP)?

Well, yes. There is always some form of it on TCP.

But your HTTP/2 reverse proxy won't block fast responses on slow responses.

As long as your WebSocket reverse proxy does the same, you're fine.

(And same stipulation for your client.)