Historically universities began as church schools, and (more or less) existed to educate priests in persuasion and nobility in leadership.

You might think that's archaic, but that's still the primary purpose of the Ivy League in the US and the older equivalents around the world. The caste system works slightly differently, with priests (persuaders and marketers) replaced by economists, lawyers, and politicians, and nobility (doers with financial/political agency) by CEOs, financiers, and oligarchs. But it's recognisably the same idea.

There are also supporting castes - the military, which has its own pipeline, and researchers/technicians, which are a weird hybrid caste. Some have limited political agency - which peaked around fifty years ago, and has been declining since - but most are just worker bees.

The idea that universities are there for personal and cultural intellectual development is relatively recent, and much more tentative. There's still a lot of hostility to it because the primary purpose of the system is to maintain power differentials, not to erode them.

The point being that the modern system is the vector sum of at least four different competing trends. There's political hierarchy, there's increasing financialisation of assets and processes (which actually conflicts with research and education), there's a need for workers who are accredited and educated enough, but not too educated and independent-minded, and there are the personal expectations of students, which depend on personality, talent, and acculturation.

There isn't a stable solution for this problem.

A recent trend is the availability of university-level teaching outside of universities. Textbook piracy, YouTube videos, and AI are all making it much easier for motivated people to learn - pretty much anything.

I'm not convinced the formal system is sustainable. But it's clear current ideas about employment aren't sustainable either. So there's going to be a period of complete chaos, and - at best - some new system of semi-formal self-motivated open education is going to replace what we have now, perhaps with some kind of external testing and accreditation for specific skills and abilities.

I agree with all of what you said, and my personal twist is that this weird, archaic, self-preserving hierarchy is a slightly-lagging expression of our collective mindset. We're changing how we think, and that is indirectly eroding away at this status quo. I see this most vividly with how we treat kids and how fast that's changing: every generation has a much better relationship with their kids than the previous generation.