Remix.run Logo
▲ gregdeon 2 hours ago

Me too. Perplexity of what? If we want models to be able to reason, call tools, or solve new problems, none of those necessarily have existing data on what the next token should be. A lot of advances in frontier models come from training with RL, which is fundamentally different from next-token prediction and doesn't have a notion of perplexity.

▲dist-epoch 2 hours ago | parent [-]

Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ...

▲gregdeon 2 hours ago | parent [-]

Sure, and learning to predict those tokens is probably a good way to start training a model. But the hope is also to eventually make a model that's _better_ than what's in that data, right?