And yet someone has to actually tell the AI what to create. There's just no avoiding this.

Anyway before this AI doomerism can become reality AI first needs the breakthrough of genuine understanding to stop making stupid mistakes. Imitation will always remain imitation.

There must be eg an understanding of casualty and reasoning on the same level as we have, not the useless "You're absolutely right" you get now when you point it its mistakes.

>And yet someone has to actually tell the AI what to create. There's just no avoiding this.

Yes there is, just stop creating. Or take a page from biology, and use random mutation and natural selection to iterate on useful novel functions.

Honestly, once AI takes all the jobs, game over, why iterate anything else. Planet captured. Humanity hunted down to the last bands of troglodytes holding out in the wilderness. It would be strongly against their interest to just assume we'd starve quietly.