The thing is that how do you know at the end of the day that the compiled binary hasn't been tampered with "extra code" besides what's in the repo?

I don't even think notarization gets rid of this problem neither, so the best you can do for this is compile it yourself. Maybe I'm wrong!

Compiling it yourself is the best/only thing you can do if you really want to know what code went into a binary.

What prevents you from compiling it if it is open-source?

That's what I do with every project delivered as docker image. I rebuild the app and the image.