The muxer can apply a timestamp offset to a track, so while the audio stream remains untouched it starts at some offset relative to the start of the video stream.

Agree once you're compensating for drift by scaling the audio track you've pushed yourself out of remux territory, but again these aren't hard and fast rules (evidenced by the disagreement over terminology).