Problem is that ableton needs really low latency audio. Can't get that in a vm.

hmm direct pipewire sink seems to be negligeable, but you could always pcie passthrough an audio card if you need latency to be measured in microseconds.