While I mostly agree with you, there's an argument to be made that classical programming tended to eliminate the mechanical parts of work that people hate doing, while AI is removing the intellectually stimulating part.

It's not fun to think of a future where knowledge workers become irrelevant in society and we're back to just physical work and managerial elites.

I'm not really one of them, but there are many developers for whom the intellectually stimulating part is designing the system. Writing the code was boring drudgery that they hated doing. They tend to call themselves "architects" or something similar, not "programmers."

Plenty of people pay the bills with work they hate doing.