GPT-5.6 Sol sets a new SOTA on ARC-AGI-3: 7.8%

Sol is the first verified frontier model to ever beat an ARC-AGI-3 game

https://arcprize.org/results/openai-gpt-5-6

Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.

▲

bevekspldnw an hour ago | parent | next [-]

These aren’t raw base models they are the result of a ton of RLHF and various adjustments.

Bitter lesson wildly overstated in this context.

	▲	froh 14 minutes ago \| parent [-]
		rlhf = reinforcement learning from human feedback (had to look it up)

▲

altcognito 2 hours ago | parent | prev | next [-]

While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.

I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)

▲

Salgat an hour ago | parent | prev | next [-]

Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).

▲

vatsachak 2 hours ago | parent | prev | next [-]

I mean, theoretically you can solve every finitary problem with a brute force solution...

Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far

▲

emp17344 an hour ago | parent | prev | next [-]

This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.

	▲	HDThoreaun an hour ago \| parent [-]
		I thought models werent allowed tools on arc-agi?

▲

Razengan 2 hours ago | parent | prev [-]

> We are probably going to need a lot more GPUs.

Or a breakthrough in algorithms etc.

The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.

▲

ryandvm 41 minutes ago | parent | next [-]

The human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat.

The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.

	▲	dbspin 19 minutes ago \| parent [-]
		Moreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2]. [1] https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/

▲

altcognito an hour ago | parent | prev | next [-]

20 watts for inference AND training!

▲

aeyes an hour ago | parent | prev [-]

For intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes.

▲

simianwords 2 hours ago | parent | prev [-]

Very interesting. My prediction is that Mythos would outperform Sol.

Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.

At some point his claim should be fully falsified no?

▲

osti 2 hours ago | parent | next [-]

Mythos probably wouldn't, otherwise they'd have included it in their release. Next version of Mythos probably will though.

And yeah.. Reality has not been kind to LeCun.

▲

vatsachak 2 hours ago | parent [-]

Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meaningful embeddings to the point where image segmentation is just PCA. The spend on DINO cannot be more than five million (correct me if I'm wrong)

JEPA is just getting started

▲

Tenoke an hour ago | parent | next [-]

His main anti-LLM predictions have been consistently either wrong or misleading.

There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.

▲

onlyrealcuzzo an hour ago | parent [-]

His main LLM predictions have almost nothing to do with Arc AGI...

What exactly was he dead wrong about that is proven by any of this?

GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.

He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.

He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.

Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.

▲

Tenoke an hour ago | parent [-]

>He's merely said they don't think

He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.

▲

onlyrealcuzzo an hour ago | parent [-]

> He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000.

What things specifically and when?

▲

Tenoke an hour ago | parent [-]

https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu

You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.

▲

onlyrealcuzzo 42 minutes ago | parent [-]

That does not at all look cherry picked or taken out of context...

LeCun's ideas cannot be reduced to a 6 second clip...

You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...

	▲	Tenoke 36 minutes ago \| parent [-]
		You can watch the whole Lex Friedman interview, it's on youtube. It's not out of context at all. He goes on about how LLMs will never be able to do things that they do trivially. And he has just doubled down for years. Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.

▲

2 hours ago | parent | prev | next [-]

[deleted]

▲

redactsureAI 2 hours ago | parent | prev | next [-]

DINO is a transformer model?

▲

esafak 2 hours ago | parent | prev [-]

ASI is going to be here by the time Lecun gets started.

▲

Maxatar an hour ago | parent | prev | next [-]

Falsifying Yann Lecun isn't exactly a priority for anyone seriously working in this space.

▲

bevekspldnw an hour ago | parent | prev [-]

“Bro” spent most of his career in the wilderness because everybody thought ML/NN/etc were a dead end.

I’d not wager against him having at one one more break though architecture before he retires.