| ▲ | jrflo 3 hours ago |
| This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day. Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember. |
|
| ▲ | seizethecheese an hour ago | parent | next [-] |
| The trend is only against a corpus of code. It’s entirely possible that the results are largely due to more intensive pretraining on code. I made a post here of a model training where code proportion is increased and you see corresponding perplexity drop https://micjm.com/blog/lowering-perplexity-without-scaling/ |
| |
| ▲ | Jepacor 13 minutes ago | parent [-] | | How do you know it's a corpus of code? I saw nothing describing what exactly their dataset is composed of in the post. | | |
| ▲ | seizethecheese 11 minutes ago | parent [-] | | Footnote from OpenAI blog post: 3. The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot). |
|
|
|
| ▲ | Marha01 2 hours ago | parent | prev | next [-] |
| > Also, I wonder what happened with GPT 4.5? It was a massive non-CoT model (for that time) that they trained as an experiment and served for a while before shelving it probably due to high inference costs. |
| |
| ▲ | arthur-st an hour ago | parent | next [-] | | It was likely a Fable-style oversized CoT model, that likely was ahead of its time with what the performance of their inference stack was at the time. | | |
| ▲ | cubefox 25 minutes ago | parent [-] | | No, it was supposed to be GPT-5 (Orion), the GPT-4 successor. It was very large but had disappointing performance gains, which it was released only much later under the name GPT-4.5. RLVR (reasoning) was not invented yet. What later became the model actually released as "GPT-5" was a different model with reasoning. |
| |
| ▲ | schleck8 an hour ago | parent | prev [-] | | Fun times, GPT 4.5 cost $75 per million input tokens and $150 per million out. For comparison, Fable 5.2 is $8 in, $40 out and cached input is only $0.20 |
|
|
| ▲ | croemer 2 hours ago | parent | prev | next [-] |
| > I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL We don't know if it's still important or not. Cross-entropy going down could just be a correlate of whatever else matters. The graph doesn't show causation: that perplexity is what makes them useful. |
| |
| ▲ | jonahx 2 hours ago | parent [-] | | The idea that compression ~ intelligence has other support: http://prize.hutter1.net/ | | |
| ▲ | AnotherGoodName an hour ago | parent | next [-] | | It goes beyond that prize. It’s also so trivial to state; If you consider intelligence as ‘being able to make directed decisions towards goals’ then better decisions are made with better predictions of the outcomes of your decisions so you better direct towards that goal (compression is mathematically just prediction at heart). The only way to argue that intelligence is not very trivially the exact same thing as compression is to argue another definition of intelligence. Which is fine but in the AI community we all 100% define intelligence as the above so there’s no question it’s the same. As in it’s not up for debate in the slightest unless someone is playing semantics. The hutter prize is just an acknowledgement of an already well known fact. | | |
| ▲ | davrosthedalek 26 minutes ago | parent [-] | | Being able to predict the future is clearly a form on intelligence, but I think for useful agentic action you need more. Don't get me wrong, being able to score possible actions by accurate predictions of their outcome is tremendously helpful. But you also need to be able to generate possible actions. That works well if you have finite options, or the options are continuous and you can efficiently optimize in your space. But in general, you need to have the inverse: Given an outcome, what action leads to this with the highest probability. | | |
| ▲ | StevenWaterman 8 minutes ago | parent [-] | | If we lean fully into it being a theoretical proof, then "i can predict the future" means I can predict what a sufficiently-good-at-generating-ideas person would come up with when asked to generate ideas |
|
| |
| ▲ | JavGull 15 minutes ago | parent | prev [-] | | In retrospect Silicon Valley is a great documentary! |
|
|
|
| ▲ | Xcelerate 2 hours ago | parent | prev | next [-] |
| > I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL From an information theory perspective, cross-entropy loss (log loss) seems to me to be about as pure as it gets in terms of capturing the notion of whether prediction "succeeded". Even Solomonoff Induction minimizes log loss. Would be curious to hear others' perspectives though on why another function might be better for universal prediction purposes. |
| |
| ▲ | pishpash 2 hours ago | parent [-] | | There are more general information measures, Rényi entropy, von Neumann entropy, etc. for different system setups, but for token-by-token information learning and prediction within classical ergodic processes Shannon entropy is where it's at. |
|
|
| ▲ | derac 3 hours ago | parent | prev | next [-] |
| I got the feeling it was too large and inefficient at the time. |
|
| ▲ | bscphil an hour ago | parent | prev | next [-] |
| If I had to guess, RL increases perplexity, but the metric is being evaluated on only their base (pre-trained) models. Highly efficient text prediction in the base model still correlates strongly with model intelligence, because there's nothing else the model could realistically learn during pre-training. Perplexity thus tracks the evolution of model intelligence over time very closely, even though (maybe) it isn't the largest component of intelligence at the end of the day. |
|
| ▲ | cubefox 28 minutes ago | parent | prev | next [-] |
| > Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember. GPT-4.5 was originally internally called Orion and was supposed to become "GPT-5". It was a huge pretrained model. However, as reported by The Information and a few other sources, the results on measured performance were disappointing. Apparently pre-training alone was leading to diminishing returns, at least with standard RLHF. This is why they didn't release it initially, only much later under the downgraded name "GPT-4.5", and only for a short period of time. Later they invented RLVR (chain-of-thaught reasoning via reinforcement learning), a post-training technique, which dramatically increased model performance, without affecting perplexity. The first model with RLVR was o1. The successor to o3 was then called GPT-5. |
|
| ▲ | londons_explore 2 hours ago | parent | prev | next [-] |
| > the increasing use of RL RL, if it works well, could theoretically reduce the perplexity too. |
| |
| ▲ | ACCount39 2 hours ago | parent [-] | | It could indeed, in theory. But in practice, perplexity-as-measured rises due to "controlled mode collapse" nature of RL training. Things that are useful for base model "next token prediction" can be useless or even actively harmful (should I elaborate?) for CoT, so the model ends up suppressing them under RL pressure. Likewise, things that are useful for CoT can harm base model next token prediction. But I bring up "perplexity-as-measured", "controlled collapse" and "suppression" because I don't believe that this "ruins" the model's underlying prediction capabilities. Generally, it merely redirects them - brings them more in line with what we care about. It may well be that with a more elaborate training and measurement setup, RL would indeed improve the "perplexity proxy" metric. |
|
|
| ▲ | pishpash 2 hours ago | parent | prev [-] |
| Cross-entropy is a generic performance metric. It just happens to be a loss function for model training but it isn't tied to model training. |