| ▲ | stevenhuang an hour ago | |
To effectively pretend to be a fool, the model needs to be smarter than the fool. That's why perplexity decreases. The meta game in pretraining is it correctly assumes the context/mind state of the writer of whatever material under test. To do this effectively requires further modeling and understanding, which is a strong predictor of capability and is a much more difficult task than simply being bad at it. | ||