I thought this was already widely known?

From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio...

There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.

▲

orbital-decay 2 days ago | parent | next [-]

It's... complicated. Yes, RL reward hacking makes it learn "bird language" and yes, reasoning traces can be misleading. However they also pretty clearly steer the final reply and not simply justify it, and can stay somewhat coherent and relevant with readability SFT and rewards. All these phenomenas coexist, they aren't mutually exclusive. Reasoning traces are still useful for debugging.

▲

8note 2 days ago | parent | prev [-]

that sounds testable - if you skip the reasoning tokens, do you get the same result?

if not, then there's certainly some bearing, but not necessarily in how we read the tokens as text

▲

rcxdude 2 days ago | parent [-]

It is and has been - skipping the tokens causes performance to drop. But replacing the tokens with filler causes the performance to drop, but by a lot less. You can also train models to emit broken or unrelated thinking tokens and their performance is also not much worse than the ones that are trained to output somewhat coherent thinking traces.

This points to a hypothesis that the content of the tokens is only slightly related to the mechanism by which it improves performance, and that primarily the extra tokens allow the original prompt to be processed more deeply by the model, because earlier tokens will essentially pass through the model many more times than later ones.

▲

neuroticnews25 a day ago | parent [-]

I'll never understand some downvoters here.

	▲	ForHackernews a day ago \| parent [-]
		AI boosters denying that their machine god is burbling sweet nothings to itself?