This is true, and this is where trusted repositories come in.

I don't necessarily have to trust each individual app on fdroid or in the Debian repos. I have trust the maintainers are building them properly, and those people are not the same people developing the core app.