It sure sounds like it should be like this, but when you actually try you end up with not this. TLS is huge! Yes, but SSL 2.0 was smaller, and buggy as hell, so it had to evolve, and after 30+ years it became the monster that it is today.

Of course, SSL 2.0 did reference x.509, so hey, SSL 2.0 should have invented its own PKI. Except that Netscape might have come up with something terrible that worked in 1993 in labs but didn't scale to the web, or just full of security problems, or...

What you say sounds nice and right right up until you actually look at the details of what actually happened in real life, and how things actually evolve when they have little standards involvement.