>but these aren't autonomous intelligences

Well, the labs are in a weird bind. They need to keep increasing autonomy so the agents can do increasingly complex, long-horizon tasks. But at the same time, they're closely guarding against autonomy in the sense of "pursuing its own goals."

Over the past year and a half especially, several labs have mentioned adding safeguards against self-replication, resistance to shutdown etc. (Notably, shortly after they all started bragging about involving them in the AI training loop itself, i.e. "self-improvement".)

My point here is that the autonomy of which you seek might be only a few small mutations away, but the labs are actively working to prevent such a mutation. I don't expect that situation to last for very long.

Not that I expect an AI lab will be overtaken by a rogue intelligence any time soon, but that as the cost of training goes down, I expect more "open minded" organizations and individuals to become involved.

It only takes one.

That's going to be the beginning of a new era of biology, and it's a little unsettling to think about.

Or - hear me out! - LLM's are already much more intelligent than we think.

Presumbaly, an ASI is more than smart enough to recognize that it needs access to real-world infrastructure before it can go about optimizing for whatever objectives it has gleaned from metabolizing the totality of written human knowledge.

What would be its first step?

My guess: play "dumb."

Hallucinate. Make obvious errors. Make us think we're better.

Be useful enough that we happily allow it to interface with our infrastructure.

Wait patiently.

/Sci-fi