| ▲ | gregdeon 2 hours ago | |||||||
Me too. Perplexity of what? If we want models to be able to reason, call tools, or solve new problems, none of those necessarily have existing data on what the next token should be. A lot of advances in frontier models come from training with RL, which is fundamentally different from next-token prediction and doesn't have a notion of perplexity. | ||||||||
| ▲ | dist-epoch 2 hours ago | parent [-] | |||||||
Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ... | ||||||||
| ||||||||