Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.

These aren’t raw base models they are the result of a ton of RLHF and various adjustments.

Bitter lesson wildly overstated in this context.

While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.

I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)

Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).

I mean, theoretically you can solve every finitary problem with a brute force solution...

Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far

This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.

> We are probably going to need a lot more GPUs.

Or a breakthrough in algorithms etc.

The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.