Remix.run Logo
▲ XenophileJKO 3 hours ago

This feels fundamentally wrong. As we move up the intelligence arc, I would expect perplexity to increase in general and only decrease on the pre train.

I would expect the perplexity on the original pre train to increase as reinforcement learning on verifiable domains teaches the model lessons that did not exist in the pre train. In addition alignment and other concerns, unless the are consistently enforced in pre train data quality will also move perplexity up.

▲athrowaway3z 2 hours ago | parent | next [-]

This feels fundamentally right to me.

The best compression you're going to achieve is when you best understand(i.e. encode) the truth of things.

Perhaps you're thinking of that too narrowly - it is not that the model gets good at understanding how a bridge stays upright; it has to also understand the truth of how a specific person might understand it, and how that specific person might put it into words.

Do that right, as slight deviation of some understanding of how humans respond in general instead of encoding for a single person, etc, etc, etc - and you reach optimal compression.

It's more efficient for the model to encode _how_ an idea is wrong, than it is to encode the wrong idea.

Its how/why they detect when they're being evaluated.

▲SsgMshdPotatoes 2 hours ago | parent | prev | next [-]

I think the perplexity is measured on the pre-train checkpoints.

▲stevenhuang an hour ago | parent | prev [-]

To effectively pretend to be a fool, the model needs to be smarter than the fool. That's why perplexity decreases.

The meta game in pretraining is it correctly assumes the context/mind state of the writer of whatever material under test. To do this effectively requires further modeling and understanding, which is a strong predictor of capability and is a much more difficult task than simply being bad at it.