Couldn’t you just monitor the inbound traffic and associate the packets to the connections? Doing your own TCP seems silly.

Yeah, some sort of packet mirroring setup (eg. in iptables or at the switch level) + packet capture tool should be enough. Then you just need to join the data from the packet capture program/machine with your load balancer, using src ip + port + time.