I tried to stop this a while ago and searching the internet did not yield great results, so I ended up creating a Shortcut that runs when the Music app opens to close it again. This actually worked well until that time I wanted to use the Music app for real.

So annoying and not great UX from Apple imo. Thanks for this.