Not an expert, so I could be wrong, but my understanding is that you could just copy those bytes directly from the wire to your memory and treat those as the Arrow payload you're expecting it to be.

You still have to transfer the data, but you remove the need for a transformation before writing to the wire, and a transformation when reading from the wire.