Remix.run Logo
▲ dist-epoch 2 hours ago

Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ...

▲gregdeon 2 hours ago | parent [-]

Sure, and learning to predict those tokens is probably a good way to start training a model. But the hope is also to eventually make a model that's _better_ than what's in that data, right?