Does AI process maybe in English?

vorticalbox 6 hours ago | parent | next [-]

Internally I don’t think anyone knows.

I read an article once that talk about this the first few layers are like a translation layers then middle is thinking and the output layers translate it back.

It’s pretty interesting actually, you can cut out the middle layers and stack them back to back and that increases the models performance without much memory increase.

If you want to try it out you can get a LLM with no tools and pass it base64 and it will understand what’s in the message

▲

alex43578 6 hours ago | parent | prev | next [-]

It seems a ton of the chain of thought tokens are English. It just processes weights, though, not tokens - maybe compute efficiency for using "cached" tokens/weights or it more efficiently learned to process English tokens by virtue of their greater weight in the corpus.

▲

gundugi-man 6 hours ago | parent | prev [-]

In Deepseek app you can see its reasoning tokens.

If I ask something in my native language (Korean), the model often switches into English in the middle and translate the answer into Korean.

But not sure about what happens below the token, in embedding space or per decoder layers.

▲

Andrex 6 hours ago | parent [-]

Wonder if this is the final step in English becoming true lingua franca. It's what every single AI will be using internally.

	▲	wongarsu 5 hours ago \| parent [-]
		I don't think the concept of natural language reasoning tokens is here to stay Current open source LLMs have an emergent structure where the first couple layers transform the input into some internal representation. Then the middle does multiple reasoning steps, and the end transforms it back to one output token. You can repeat the middle to get a stronger, slower model, so it is abstract reasoning. And Anthropic had a blog post about actually looking at what the middle does and mapping it back to concept for inspection With that in mind, what's the point of reasoning tokens? They throw rich internal state away, for the benefit of inspectability and variable-length reasoning. But now that we understand the internal structure better we can dynamically vary how many loops the model spins through those middle layers, and still get enough insight into what's happening by decoding internal state. This takes some reengineering of model architecture and inference engines (especially to still get good batch performance), but the potential benefits seem to big to ignore