Yep this is a glimpse into the future of 500+ t/s, which is in my opinion the next big thing that validates Jevon's paradox (the models are already smart enough)

I think the glimpse that is there will be exclusive access. So much for the open in openAI. If this technology really transforms society in the ways expected with inequality an unavoidable consequence equal access should be required like internet access was (isp can’t give preference to specific user traffic)

Faster tokens = more reasoning loops, so it can actually make the models smarter as well.

“Smart enough” really depends on how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet, IMO.

Most of the frontier models can, when prompted and tooled correctly, do a lot of “reasoning” tasks that amount to resolving how the user has explained a particular widely known paradigm.

The more difficult and obscure the issues you provide them with, the faster you notice them reward hacking by altering the criteria until they are no longer attempting to solve the problem. Using “advisor” style loops helps hold this off at the cost of tokens, but there is still a fairly short limit at which they will essentially give up if they can’t find all of the necessary information - sometimes the issue is actually worse if they find a small amount of information instead of nothing - they’ll extrapolate from that tiny piece of data and generate plausible-sounding hallucinations almost every time.

And god forbid your problem involves doing something a different way than the majority of people do it. Unless you can write a full spec on it, the models will repeatedly spiral back into adjusting everything about your problem until it matches one of the most popular approaches in their training data.

[dead]