I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…

▲

thinking_cactus 43 minutes ago | parent | next [-]

How can they do that? Does state propagate between each token output?

That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?

▲

markasoftware an hour ago | parent | prev | next [-]

Yes dots increases precision, but not nearly the same increase in precision as having actual useful reasoning in the CoT

▲

gr_norm 3 hours ago | parent | prev [-]

Link to this study?

▲

khalic 3 hours ago | parent [-]

[2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

https://arxiv.org/abs/2404.15758

	▲	comex 13 minutes ago \| parent \| next [-]
		The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all. Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2. Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT. Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
	▲	stymaar 2 hours ago \| parent \| prev [-]
		Also 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! https://arxiv.org/abs/2504.09762