Open source already has the "the collective data of humanity as a whole". That's the LLM leg. Now focus on what they train their models on for RL. Do other models have less code for training coding? for math? Those are pretty much the only cases where we can have reliable validation. Then, what's the next step. So many branches: legal contracs? etc. So much more obscure detail, which is why (I'd guess) so many startups exist in these areas.

AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.