Even if I had the skills to confirm the code is secure, how could I know that this is the code running on my phone, without also having the skills to build and deploy it from source?

Also, you need to make sure that the installation process does not insert a backdoor into the code you built from source.