Remix.run Logo
▲ tensor 3 hours ago

I'm a bit skeptical about the claim that particular abilities "emerge" at given perplexities. For instance, there are smaller models that are specifically trained on math, and I imagine those models have much higher general perplexity on other tasks.

On the other hand, assuming you have appropriate training data for a task, then yes obviously perplexity is going to measure performance on that task. Being able to predict the next task token is obviously going to be a near perfect correlation with accuracy on the task.

▲namr2000 3 hours ago | parent | next [-]

While intuitively it would make sense that smaller models specifically trained on math perform better than large generalist models, this has empirically proven to be untrue.

Deepmind was proving math theorems using a model specifically trained on math, and that was very quickly surpassed by generalist LLMs. It's presently not well understood why this is the case.

▲rednb 2 hours ago | parent | next [-]

I'm surprised that this is surprising for the industry. All knowledge proceeds from the same wisdom. The more fields you master at a very advanced level (eg. STEM, philosophy, marketing, etc...). The more you notice that the same basic principles drive seemingly unrelated fields.

I am not simply talking about applying a mathematical technique commonly used in one field to another. At a deeper level, you always get back to playing the same game in a different form. For example, sales engineering has a lot in common with cybersecurity.

And this is a good example, as it's extremely rare to find someone with advanced expertise in both fields, since they typically attract two radically different kind of people.

For this reason, where the output expected is not mechanical, i'm not surprised at all that generalist models are doing better than specialized ones for open ended questions and frontier work.

▲theptip 2 hours ago | parent [-]

This is definitely a conjecture; it gets to how much SGD will generalize out of the training distribution.

If pretraining alone was sufficient then nobody would do RL. And, heavy RLVR on coding/math is why these models are savant-level in those domains, yet not superhuman at all fuzzy reasoning and practical wisdom tasks that are only represented in the pretraining corpus, and don’t have an RL target.

▲kypro 10 minutes ago | parent | prev | next [-]

If I were to guess it's because a model trained purely on maths wouldn't be as likely to create such diverse and deep layers of abstractions as it would need to if it were being trained to predict something as complex as the real world.

A maths model trained only on maths would only learn how to predict something if a pattern was exposed often enough and was a useful enough predictive signal. While I'm sure it could all the same patterns and abstractions with a maths dataset alone, it would depend far more on that dataset being exceptionally well designed.

As an example imagine a self-driving car. A self-driving car needs to understand road signage. You could train a self-driving car with all of the signs in the US and I'd suggest that it wouldn't be as good at predicting signs in the real-world as an AI which in addition to being trained on road signs, could also read text and understands material science. Why? Because in the real world signs can be damaged and distorted. An AI that could understand the ways in which signs might distort and can understand that a four letter word beginning with an S and ending P is probably "STOP" even the T and O are ineligible. Of course, an AI with a rich enough dataset containing damaged signs might eventually get equally as good, but perhaps would never quite understand the patterns that underlie the patterns - it would recognise all of the way a stop sign can be damaged and distort, but not why the distortion happens or what the letters on the sign are and how they relate.

▲hn_throwaway_99 25 minutes ago | parent | prev | next [-]

Correct me if I'm wrong but I thought there were still a number of cases where models trained for specific tasks perform much better than frontier generalized LLMs. E.g. Stockfish and KataGo perform much better at chess and go than the best frontier models (obviously I don't know what the internal models are capable of in these realms).

▲davidguetta 2 hours ago | parent | prev | next [-]

general training remove noise

▲magicalhippo 2 hours ago | parent [-]

I wonder if it's also that it gives them a better sense of analogies. In my experience small LLMs do really poorly with analogies, but the large frontier models seem to get them much, much better.

In math at least I often use analogies to help solve problems, like "ah maybe if I think of this as a vector I might use linear algebra here" and such.

▲trash_cat 2 hours ago | parent | prev [-]

Sounds like a bitter lesson.

▲theptip 2 hours ago | parent | prev | next [-]

Yeah, my model was that pretraining, RL, and test-time all had their own scaling returns, so just looking at pretraining loss perhaps oversimplifies?

I could believe that your RL/test-time scaling curves are strongly constrained by the pretraining loss and so in some sense that makes the max/optimal performance “latent”.

But its not clear that pretraining loss alone gives enough information on capabilities, particularly as we get closer to dangerous thresholds in some dimensions. (Eg CBRN capabilities, to take a mundane harm that’s currently impacting us directly.)

I’d like to see the scaling curves here; intuitively I’d guess you get more return from RL/test time, the smarter the base pretrained checkpoint is; therefore the error bars on capability/perplexity should be getting wider on this chart.

▲obblekk 3 hours ago | parent | prev | next [-]

Yes training on math would give you spiky abilities on math. But in this case the dataset they measure perplexity is ood from training so lower perplexity on it indicates a general increase in intelligence across domains.

But there’s definitely spikiness and it’s very possible they’ve trained more on math than other domains.

▲rbn3 2 hours ago | parent | prev | next [-]

there's a whole field of science that concerns itself with emergence as a phenomenon, where complex systems suddenly exhibit new dynamics once they cross a certain threshold. stuart kauffman's "the origins of order" is a really nice read if you want to get into the early stages of that kind of science. a lot of work done in that area by biologists trying to explain the origin of life/self-organizing matter. a more recent term from AI research that describes a specific critical change in a network past a certain threshold is "grokking".

▲zodwick 3 hours ago | parent | prev | next [-]

why would you be skeptical given that the claims seem to hold up in real world ? Assuming they measure stuff correctly

▲nutjob2 2 hours ago | parent [-]

Because correlation doesn't prove anything? The "real world" might be telling you something but maybe not what you think it is.

▲gregdeon 2 hours ago | parent | prev [-]

Me too. Perplexity of what? If we want models to be able to reason, call tools, or solve new problems, none of those necessarily have existing data on what the next token should be. A lot of advances in frontier models come from training with RL, which is fundamentally different from next-token prediction and doesn't have a notion of perplexity.

▲dist-epoch 2 hours ago | parent [-]

Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ...

▲gregdeon 2 hours ago | parent [-]

Sure, and learning to predict those tokens is probably a good way to start training a model. But the hope is also to eventually make a model that's _better_ than what's in that data, right?