Remix.run Logo
▲ joe_the_user an hour ago

Quized ChatGPT about the article...

I had trouble seeing it in the article (it's hidden in footnote 3) but it seems perplexity/log-loss is tracked using the model's ability to track an unreleased code base of OpenAI.

The log-loss involved in code has declined more for code than prose in all of the models but has supposedly declined everywhere. [1]

At the same time, the decline of the latest models on a common sense benchmark might be meaningful: https://news.ycombinator.com/item?id=50588916

[1] https://openreview.net/pdf?id=SHMj84U5SH