What happens if there are no more humans in the loop, and instead of building on human work, models have to build on their own work? Would we get stuck in a local maxima?

When AI labs talk about “long-horizon tasks”, they mean tasks on the order of a few days. But what about keeping coherence across decades, like humans can? Models are not trained or tested on the ability to do that, and it seems unlikely that capability would naturally exist.

I'm pretty sure that if you put a few LLMs, agents, or anything into a room together without outside access, and get them to learn off of each other, you'll end up with gibberish, nonsensical garbage and definitely nothing magical will happen. Delirium, oblivion, dementia and so on...

Coherence is a given... What we lose over decades is alignment.