It'd be a lot of trouble to interfere with the source, yes.

I think the release files is the place they could most easily tamper - generally they're stored on Github infra so the files could be changed, and the checksum on the download page also altered (or different files and different checksums provided to different people if targeted).

Unless the builds are totally reproducible it'd be tricky to catch.

Possible, yes, but pretty damming to Microsoft's reputation if proof that their infrastructure has been compromised and anyone realizes it's happening. This sort of thing killed Sourceforge when they started shipping adware bundled into installers of the programs they distributed.