| ▲ | synapsomorphy an hour ago | |
The big part still being left out, to me, is the size of the model itself. Minimum descriptor length and Kolmogorov complexity are always size(compressed data + compressor). Right now we're mostly just looking at the first part, but it's trivial to smuggle in things in the second part that can make the first part appear as low as you like. | ||
| ▲ | nsagent 19 minutes ago | parent [-] | |
You're right. If they really wanted to address the problem of coordination, then they would look at better measures like MDL. The cynical part of me thinks that the main reason for their perplexity improvements on this held out dataset is due to collecting more and more data to train on, so the held out dataset is effectively becoming more and more in-domain. Without trying to estimate joint prequential MDL, it's nearly impossible to tell. Admittedly, they can't go back and compute the prequential MDL of their previously trained models, but they could look at normalized compression distance using a fixed LLM as the reference as a reasonable alternative. Though of course, just saying perplexity correlates well with their empirical assessment of intelligence certainly makes them look like they are accelerating (which again, my cynical side tells me is really the goal of this blog post; to set themselves up for the IPO they pushed off). | ||