I've got this setup recently via pipewire! With something like barrier for kvm, it's great!

I have an alias on each side -- one gets a new source, the other a new sink.

  audio_recv='pactl load-module module-native-protocol-tcp port=4656 listen=0.0.0.0'
  audio_send='pactl load-module module-tunnel-sink server=target:4656'