I imagine we know we have reached AGI when a technocrat stops sharing their AI. It goes from being something they can sell to something that they don’t want to share. Instead they ask it how they can dominate the world and be the first trillionaire and how they can stop anyone else acquiring an AGI etc.

This even works at a smaller not so general level: imagine that one of today’s popular code models improved to the point it is better (narrowly at programming) than a human. Suddenly the owner shouldn’t sell it to everyone: instead they should pivot and make software that outcompetes anything a human can make. So it doesn’t just replace humans making software but also replaces the programs that people made…