That‘s quite a big leap, and sounds like a philosophical question. But many philosophers like late Wittgenstein or Heidegger disagreed with this idea. On more practical terms, maybe you‘ve experienced the following: You read a manual of a device on how to do something with it; but only actually using it for a few times gives you the intuition on how to use it _well_. Text is just very lossy, because not every aspect of the world, and factors in your personal use, are described. Many people rather watch YouTube videos for eg repairs. But those are very lossy as well - they don’t cover the edge cases usually. And there is often just no video on the repair you need to do.

BTW, have you ever tried ChatGPT for advice on home improvement? It sucks _hard_ sometimes, hallucinating advice that doesn’t make any sense. And making up tools that don’t exist. There‘s no real commonsense to be had from it. Because it’s all just pieces of text that fight with each other for being the next token.

When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. There is no real understanding of how Swift („the world“) works.

▲

timschmidt 16 hours ago | parent | next [-]

None of your examples refute the direct evidence of internal world model building which has been demonstrated (for example: https://adamkarvonen.github.io/machine_learning/2024/01/03/c... ).

Instead you have retreated to qualia like "well" and "sucks hard".

> hallucinating

Literally every human memory. They may seem tangible to you, but they're all in your head. The result of neurons behaving in ways which have directly inspired ML algorithms for nearly a century.

Further, history is rife with examples of humans learning from books and other written words. And also of humans thinking themselves special and unique in ways we are not.

> When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result.

Yep. And humans often need to reference the documentation to get details right as well.

▲

manmal 13 hours ago | parent [-]

Unfortunately we can’t know at this point whether transformers really understand chess, or just go on a textual representation of good moves in their training data. They are pretty good players, but far from the quality of specialized chess bots. Can you please explain how we can discern that GPT-2 in this instance really built a model of the board?

Regarding qualia, that’s ok on HN.

Regarding humans - yes, humans also hallucinate. Sounds a bit like whataboutism in this context though.

▲

timschmidt 13 hours ago | parent [-]

> Can you please explain how we can discern that GPT-2 in this instance really built a model of the board?

Read the article. It's very clear. To quote it:

"Next, I wanted to see if my model could accurately track the state of the board. A quick overview of linear probes: We can take the internal activations of a model as it’s predicting the next token, and train a linear model to take the model’s activations as inputs and predict board state as output. Because a linear probe is very simple, we can have confidence that it reflects the model’s internal knowledge rather than the capacity of the probe itself."

If the article doesn't satisfy your curiosity, you can continue with the academic paper it links to: https://arxiv.org/abs/2403.15498v2

See also Anthropic's research: https://www.anthropic.com/research/mapping-mind-language-mod...

If that's not enough, you might explore https://www.amazon.com/Thought-Language-Lev-S-Vygotsky/dp/02...

or https://www.amazon.com/dp/0156482401 to better connect language and world models in your understanding.

▲

manmal 12 hours ago | parent [-]

Thanks for putting these sources together. It’s impressive that they got to this level of accuracy.

And is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible?

	▲	timschmidt 12 hours ago \| parent [-]
		> is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible? Overfitting happens, even in humans. Have you ever met a scientist? My points have been only that 1: language encodes a symbolic model of the world, and 2: training on enough of it results in a representation of that model within the LLM. Exhaustiveness and accuracy of that internal world model exist on a spectrum with many variables like model size, training corpus and regimen, etc. As is also the case with humans.

▲

tsunamifury 16 hours ago | parent | prev [-]

[flagged]