You could that with a much saner approach like DANE.
Not back when SSL and the PKI ecosystem was developed.