Remix.run Logo
▲ Ohentis 5 hours ago

No that would be a pretty terrible next token predictor. It would perfectly match the training data, but the training data is just a sample of what we are really trying to predict: human language, knowledge, and thinking, which is inherently probabilistic. Perfectly matching the training data is a pretty good sign that you've overfit.