I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of?

Our current AI is more like a fancy Google search than some kind of machine God.

"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence.

How do we get to ASI? That's what recursive self-improvement is about.

If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.

We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.