I think the counter arguments to doom are more social and economic. If we make AI smarter than ourselves it's likely people will still have jobs/income and a higher standard of living in the same way that in invention of the tractor didn't lead to millions of unemployed potato diggers and a potato tycoon with a million tons of spare potatoes.

> likely people will still have jobs/

doing what though? in a really literal sense, if an LLM can do RLVR, that's optimization over the space of algorithms, and so anything with a measurable goal at the end can be incentivized. if it can be measurably finished, it can be automated.

there are exceptions to this, like psychology, preschool teachers, and AI safety, that cant be 100% fully automated. in the case of Ai safety, the core motivation for it is to prevent the otherwise hyper-consequentialist goal-orented training algorithms (and mesa-optimizers they now find) from doing 'bad' things by instilling some 'values' into these machines. the ideas of 'good/bad' and 'values' cant be automated away, and I think you can prove that its unautomatable using EY's orthogonality thesis.

I doubt these few unautomatable jobs can keep everyone afloat though. but maybe a world in which 10% of the population has a degree in psychology wouldn't be that bad