All current era models overthink as it's a product of their RL incentives (or distillation of models with them...)

From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like:

Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively.

That is fantastic for SWE benchmarks and autonomous agents. It also naturally creates pathologies:

under-answering is expensive; over-answering is cheap.

The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models.

For a complex maths problem, Sol reasoned in 367 tokens before working:

  We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111. Count binary strings length16 weight8 avoiding 111. Need N mod1000. Compute stepwise perhaps runs of 1 length max2. Count via runs.
  
  [... cut in half for HN readability ... ]

  Check interpretation potentially "no person sits next to two other people": no seated person's chair adjacent to two occupied chairs. Exactly no three consecutive chairs selected. yes.
  
  Need reason step by step final boxed 907. Explain runs. Ensure people each select chair distinct subset (given subset count). Let's present.

I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).

[1] https://stolen-thoughts.com/

For a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good.

I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.

I believe this is what Meta is doing as they started recording their SWE screens some time ago.

For coding, this is very interesting because the same incentives were present for humans before AI. Tech companies which had a culture of rewarding complexity would see huge Pull Requests and a lot of unnecessary complexity. I've worked in companies which would require a thousand lines of code to implement a feature which would require only a hundred or so lines at a different company. The shorter one was more reliable too. Code begets more code. The incentives created by the company culture had a massive impact... And the culture was heavily determined by whether or not the company had a market monopoly. More monopoly power -> more unnecessary complexity (presumably so that employees could achieve better lock-in/job security through the increased need to manage that complexity; in any case, the company could comfortably afford and it did not present an existential risk as it would in a startup environment).

So it's not surprising that the same dynamics are at play with AI. Now, because code is being churned out so rapidly, the effects have become much more obvious (it took me years to figure this out, but now managers can observe this same effect play out in months); many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...

Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% (I.e. 5% more tokens to solve the same problem) and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (purely as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... The insider's term for this is 'technical debt' for multiple reasons; including the fact that it compounds like normal debt. Now factor in the monopolistic tendencies of those tech markets... It's a real bottomless gold mine.

Revenue from downstream corporate users comes in regardless of compounding code complexity and slower pace of delivery; those companies just keep hiring more people, spending more on tokens. Swallowing up these massive diminishing returns like an appetizer. Enshittification takes place but the downstream end user has nowhere else to go.

It's trivial for a lab to advertise themselves as being token-efficient and almost impossible for its corporate users to actually verify it.

The only real issue with that business model (possibly a fatal issue) are these open weights models which the big tech companies could use to move off the AI service platforms if the problem becomes bad enough.