Remix.run Logo
▲ nsagent 27 minutes ago

You're right. If they really wanted to address the problem of coordination, then they would look at better measures like MDL.

The cynical part of me thinks that the main reason for their perplexity improvements on this held out dataset is due to collecting more and more data to train on, so the held out dataset is effectively becoming more and more in-domain. Without trying to estimate joint prequential MDL, it's nearly impossible to tell.

Admittedly, they can't go back and compute the prequential MDL of their previously trained models, but they could look at normalized compression distance using a fixed LLM as the reference as a reasonable alternative.

Though of course, just saying perplexity correlates well with their empirical assessment of intelligence certainly makes them look like they are accelerating (which again, my cynical side tells me is really the goal of this blog post; to set themselves up for the IPO they pushed off).