Would the issue not be that you would need to trust that first connection?

Yep https://en.wikipedia.org/wiki/Trust_on_first_use