Good sci fi premise, but not at all how AGI will happen.

It’s not going to be a singularity at one moment of time. It’s not going to be instant runaway self-improvement, no matter what doomers and fetishists say.

It’s going to be gradual. We’ll see glimmers of AGI, and the “G” part will be about gradual broadening of domains and deepening of capabilities.

All of the coding harnesses are already using their own tools to self-improve, and the HITL component is getting less frequent and at higher levels of abstraction.

That’s how AGI gets here: very gradually, no hard takeoff, and nobody will be able to pinpoint when exactly it happened.

So: also no single lab with a massive advantage, no government takeovers. It’ll be a lot less dramatic than the extremes believe. IMO, of course.

So, extend what they said to cover a 50 year timespan? "Instantaneous" wasn't mentioned or implied.

The problem is... the moment someone gets it and it really solves hard problems and solves scifi level shenanigans, the given country that owns it, could gain such unfathomable lead above anyone else that it will almost surely lead to an all out war. I don't even know whether it is possible to conceal that you have such capability....

Imagine all the fear- and warmongering kingmakers and powerful individuals when they realize they have no power over anything or anyone.....so game over for them. They won't like it at all, at all.

Another problem, that in order to make it understand real life, it needs robots or humans wired into it (brain interfaces) in order to test certain things in the real world. And that is another level we know almost nothing about, at least on the surface.

PS: do these self improving harnesses even work?

Someone could have a small private breakthrough tomorrow that gives sample learning efficiency of the brain, online learning, and consolidated memories.