You're sure not vetting any byte of an executable, so building from source is safer.

Binaries or source, it's pretty much the same unless you thoroughly vet the entire source code. Malicious code isn't advertised and commented and found by looking at a couple of functions. It's carefully hidden and obfuscated.