| ▲ | dist-epoch 2 hours ago | |
Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ... | ||
| ▲ | gregdeon 2 hours ago | parent [-] | |
Sure, and learning to predict those tokens is probably a good way to start training a model. But the hope is also to eventually make a model that's _better_ than what's in that data, right? | ||