That's an interesting take. I agreed with your first paragraph, but didn't expect the conclusion.

From my perspective, the statement that these technologies are taking us to AGI is the overly confident part, particularly WRT the same lack of understanding you mentioned.

I mean, from just a purely odds perspective, what are the chances that human intelligence is, of all things, a simple next-token predictor?

But, beyond that, I do believe that we observably know that it's much more than that.