Remix.run Logo
▲ stevenhuang an hour ago

To effectively pretend to be a fool, the model needs to be smarter than the fool. That's why perplexity decreases.

The meta game in pretraining is it correctly assumes the context/mind state of the writer of whatever material under test. To do this effectively requires further modeling and understanding, which is a strong predictor of capability and is a much more difficult task than simply being bad at it.