Remix.run Logo
▲ MrCheeze 3 hours ago

I always had the impression that model capabilities were stagnating around GPT-5.0, and this finally broke around the release of Claude 4.5 later that year. Seems like their own graph corroborates that things were pretty stagnant then. I wonder what changed?

▲kalcode 3 hours ago | parent | next [-]

Oh oh! The models actually know what happen as well if you want to dig deeper. Pre-scale brute force training hit there limits. They broke through late 2025 due to RL training techniques that taught them to fail, fail, fail to succeed instead of spin off into a corner with nonsense.

Basically we started tuning them to be smarter and able to handle 2 to 4 hour tasks.

I believe this will hit it's limits as well. When IDK.

▲SwellJoe 2 hours ago | parent | next [-]

Given the current models from Anthropic and especially OpenAI have a hard time constructing coherent English sentences that a human can understand even while being better at code and math, I'd say they're reaching that limit; can't use a model that you can't communicate with. The Deepmind folks seem to have figured something out on that front with Neon, though, as it's both smart/capable and a clear communicator.

▲bscphil an hour ago | parent [-]

I think they're probably just RLVRed into oblivion to do well at coding, because that's what 98% of their revenue comes from. That's how you get the precise language which still feels awkward and stilted, because internally the model learns to align specific peculiar expressions to precise meanings, which are useful when you're trying to write code. There's no point in using a complete sentence when you're mostly talking to yourself; the result is intelligible, just terse to the point of absurdity. (And some models, including Google's, still have a strong tendency toward flattery thanks to RLHF. Ah well...)

Just to have a shared example of AI text, this is pretty instructive I think: https://statmodeling.stat.columbia.edu/2026/11/21/by-program...

Where a human would write "by the end of the program" the model (feels like Claude to me but idk) writes "by program end". Where a human would write "There are two scheduling considerations for program hosts." the models writes "Two timing windows worth flagging."

The latter text, "two timing windows worth flagging" is highly specific and precise if you're a model keeping track of something in the docs of a repository across contexts, but it's terrible human communication.

▲ACCount39 2 hours ago | parent | prev | next [-]

This graph specifically tries to measure pre-train perplexity. It measures the "foundation" of a given base model - before RL is applied to it.

On long term tasks, "CoT + RL" of o1/o3 beats GPT-4.5 resoundingly. But on raw base model perplexity, GPT-4.5 is a significantly better model than any of the o-series.

The curve following GPT-4.5 and o-series suggests that newer models take advantage of BOTH better base model capabilities (perplexity on held-out data) and better RL.

▲kalcode an hour ago | parent [-]

Ohh, neat thanks for the spotlighting that, I didn't know that.

▲famouswaffles 2 hours ago | parent | prev [-]

They are reporting perpexlity on pre-trains not RL-tuned models.

▲chubot 3 hours ago | parent | prev | next [-]

I'm not an expert, but I imagine it's what I wrote in a sibling comment -- training to improve agent performance (reinforcement learning with verifiable rewards).

That generates new data that's not on the Internet -- more true observations to learn from, compress, and generalize.

I think that improves the models in general, not just for agents. (And also I think it's the cause of the "Claudish" prose dialect we all started noticing)

I think that matches the timing, though I appreciate any closer analysis

▲jordanb 3 hours ago | parent | prev | next [-]

I think most of the progress in the last few years was more RF and harnesses, especially wrt (for example) claude code.

▲Wowfunhappy 3 hours ago | parent | prev [-]

[dead]