Does anyone have more insight into how chain of thought might be subverted without meaningfully impacting model performance? I’ve heard this for a while now, and I understand how information might be retained in the weights that isn’t documented in the output. But weren’t reasoning models created in the first place because they provided a performance improvement in terms of output? Is that no longer the case? If so, why are the big labs still creating reasoning models?

My understanding is that the extra token vectors generated as reasoning are still useful, but that their surface form (tokens themselves) do not necessarily reflect the underlying reasoning. i.e. reading the reasoning traces could be complete gibberish, but the hidden-dim vectors themselves still refine the latent probabilities and help in generating the correct answer.

Not an expert in LLMs, but this seems supported by the abstract of the paper cited in the above article:

  it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. [...] our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.

Sibling posts are correct -- the chain-of-thought is doing hidden computation, it has been shown in the linked papers.

If you want to see it yourself: load up Qwen 3.8 in LM Studio and watch the CoT stumble around like a drunken sailor before miraculously jumping to the correct result.

If you want an example of subversion, Anthropic has some good ones:

https://transformer-circuits.pub/2025/attribution-graphs/bio...

https://transformer-circuits.pub/2025/attribution-graphs/bio...

Before RL we typically SFT on human reasoning traces. This makes the reasoning traces somewhat coherent and the model trains faster.

But you don’t have to do that. You can skip straight to RL. If you do, the model will generate complete garbage reasoning traces before generating the correct answer. In fact, if you add a coherence reward to the reasoning trace, the model will perform worse (since you’re now diluting the correctness reward).

I'm not sure anyone meaningfully understands it: "Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens" https://arxiv.org/html/2505.13775v3

> More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, achieve performance largely comparable to those trained on correct traces.

https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-...

> *TLDR*: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)