| ▲ | tyre 2 days ago |
| There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages. It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case. I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling. [0]: https://www.anthropic.com/research/tracing-thoughts-language... |
|
| ▲ | manmal 2 days ago | parent | next [-] |
| > Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages I thought that’s the basic premise of how transformers work - they encode concepts into high dimensional space, and similar concepts will be clustered together. I don’t think it models the world, but just the texts it ingested. It’s observation and regurgitation, not understanding. I do use agents a lot (soon on my second codex subscription), so I don’t think that’s a bad thing. But I’m firmly in the “they are useful tools” camp. |
| |
| ▲ | bryanlarsen a day ago | parent [-] | | That's a model. Not a higher-order model like most humans use, but it's still a model. | | |
| ▲ | manmal a day ago | parent | next [-] | | Yes, not of the world, but of the ingested text. Almost verbatim what I wrote. | | |
| ▲ | timschmidt a day ago | parent | next [-] | | The ingested text itself contains a model of the world which we have encoded in it. That's what language is. Therefore by the transitive property... | | |
| ▲ | manmal 19 hours ago | parent [-] | | That‘s quite a big leap, and sounds like a philosophical question. But many philosophers like late Wittgenstein or Heidegger disagreed with this idea. On more practical terms, maybe you‘ve experienced the following: You read a manual of a device on how to do something with it; but only actually using it for a few times gives you the intuition on how to use it _well_. Text is just very lossy, because not every aspect of the world, and factors in your personal use, are described. Many people rather watch YouTube videos for eg repairs. But those are very lossy as well - they don’t cover the edge cases usually. And there is often just no video on the repair you need to do. BTW, have you ever tried ChatGPT for advice on home improvement? It sucks _hard_ sometimes, hallucinating advice that doesn’t make any sense. And making up tools that don’t exist. There‘s no real commonsense to be had from it. Because it’s all just pieces of text that fight with each other for being the next token. When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. There is no real understanding of how Swift („the world“) works. | | |
| ▲ | timschmidt 18 hours ago | parent | next [-] | | None of your examples refute the direct evidence of internal world model building which has been demonstrated (for example: https://adamkarvonen.github.io/machine_learning/2024/01/03/c... ). Instead you have retreated to qualia like "well" and "sucks hard". > hallucinating Literally every human memory. They may seem tangible to you, but they're all in your head. The result of neurons behaving in ways which have directly inspired ML algorithms for nearly a century. Further, history is rife with examples of humans learning from books and other written words. And also of humans thinking themselves special and unique in ways we are not. > When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. Yep. And humans often need to reference the documentation to get details right as well. | | |
| ▲ | manmal 15 hours ago | parent [-] | | Unfortunately we can’t know at this point whether transformers really understand chess, or just go on a textual representation of good moves in their training data. They are pretty good players, but far from the quality of specialized chess bots. Can you please explain how we can discern that GPT-2 in this instance really built a model of the board? Regarding qualia, that’s ok on HN. Regarding humans - yes, humans also hallucinate. Sounds a bit like whataboutism in this context though. | | |
| ▲ | timschmidt 15 hours ago | parent [-] | | > Can you please explain how we can discern that GPT-2 in this instance really built a model of the board? Read the article. It's very clear. To quote it: "Next, I wanted to see if my model could accurately track the state of the board. A quick overview of linear probes: We can take the internal activations of a model as it’s predicting the next token, and train a linear model to take the model’s activations as inputs and predict board state as output. Because a linear probe is very simple, we can have confidence that it reflects the model’s internal knowledge rather than the capacity of the probe itself." If the article doesn't satisfy your curiosity, you can continue with the academic paper it links to: https://arxiv.org/abs/2403.15498v2 See also Anthropic's research: https://www.anthropic.com/research/mapping-mind-language-mod... If that's not enough, you might explore https://www.amazon.com/Thought-Language-Lev-S-Vygotsky/dp/02... or https://www.amazon.com/dp/0156482401 to better connect language and world models in your understanding. | | |
| ▲ | manmal 14 hours ago | parent [-] | | Thanks for putting these sources together. It’s impressive that they got to this level of accuracy. And is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible? | | |
| ▲ | timschmidt 14 hours ago | parent [-] | | > is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible? Overfitting happens, even in humans. Have you ever met a scientist? My points have been only that 1: language encodes a symbolic model of the world, and 2: training on enough of it results in a representation of that model within the LLM. Exhaustiveness and accuracy of that internal world model exist on a spectrum with many variables like model size, training corpus and regimen, etc. As is also the case with humans. |
|
|
|
| |
| ▲ | tsunamifury 17 hours ago | parent | prev [-] | | [flagged] |
|
| |
| ▲ | tsunamifury 17 hours ago | parent | prev [-] | | Bruh compressing representations into linguistics is a human world model. I can’t believe how dumb ask these conversations are. Are you all so terminally nerd brained you can’t see the obvious |
| |
| ▲ | sleepyams a day ago | parent | prev | next [-] | | What does "higher-order" mean? | |
| ▲ | dgfitz a day ago | parent | prev [-] | | I believe that the M in LLM stands for model. It is a statistical model, as it always has been. |
|
|
|
| ▲ | _fizz_buzz_ a day ago | parent | prev | next [-] |
| > Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages. That doesn't seem surprising at all. My understanding is that transformers where invented exactly for the application of translations. So, concepts must be grouped together in different languages. That was originally the whole point and then turned out to be very useful for broader AI applications. |
|
| ▲ | overfeed a day ago | parent | prev | next [-] |
| > Basically an LLM translating across languages will "light up" for the same concepts across languages Which is exactly what they are trained to do. Translation models wouldn't be functional if they are unable to correlate an input to specific outputs. That some hiddel-layer neurons fire for the same concept shouldn't come as a surprise, and is a basic feature required for the core functionality. |
| |
| ▲ | balder1991 a day ago | parent [-] | | And if it is true that the language is just the last step after the answer is already conceptualized, why do models perform differently in different languages? If it was just a matter of language, they’d have the same answer but just with a broken grammar, no? | | |
| ▲ | kaibee a day ago | parent [-] | | If you suddenly had to do all your mental math in base-7, do you think you'd be just as fast and accurate as you are at math in base-10? Is that because you don't have an internal world-model of mathematics? or is it because language and world-model are dependently linked? |
|
|
|
| ▲ | bravura a day ago | parent | prev | next [-] |
| How large is a lion? Learning the size of objects using pure text analysis requires significant gymnastics. Vision demonstrates physical size more easily. Multimodal learning is important. Full stop. Purely textual learning is not sample efficient for world modeling and the optimization can get stuck in local optima that are easily escaped through multimodal evidence. ("How large are lions? inducing distributions over quantitative attributes", Elazar et al 2019) |
| |
| ▲ | EMM_386 15 hours ago | parent | next [-] | | > How large is a lion? Ask a blind person that question - they can answer it. Too many people think you need to "see" as in human sight to understand things like this. You obviously don't. The massive training data these models ingest is more than sufficient to answer this question - and not just by looking up "dimensions of a lion" in the high-dimensional space. The patterns in that space are what generates the concept of what a lion is. You don't need to physically see a lion to know those things. | |
| ▲ | latentsea a day ago | parent | prev [-] | | > How large is a lion? Twice of half of its size. | | |
| ▲ | johnisgood a day ago | parent [-] | | Can you be more specific about "size" here? (Do not tell me the definition of size though). You are not wrong though, just very incomplete. Your response is a food for thought, IMO. |
|
|
|
| ▲ | Hendrikto a day ago | parent | prev | next [-] |
| That is just how embeddings work. It does not confirm nor deny whether LLMs have a world model. |
|
| ▲ | SR2Z 2 days ago | parent | prev | next [-] |
| Right, but modeling the structure of language is a question of modeling word order and binding affinities. It's the Chinese Room thought experiment - can you get away with a form of "understanding" which is fundamentally incomplete but still produces reasonable outputs? Language in itself attempts to model the world and the processes by which it changes. Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise - but you could make a very good case, for example, that understanding the same thing in poetry gets an LLM much closer. |
| |
| ▲ | hackinthebochs 2 days ago | parent | next [-] | | LLMs aren't just modeling word co-occurrences. They are recovering the underlying structure that generates word sequences. In other words, they are modeling the world. This model is quite low fidelity, but it should be very clear that they go beyond language modeling. We all know of the pelican riding a bicycle test [1]. Here's another example of how various language models view the world [2]. At this point it's just bad faith to claim LLMs aren't modeling the world. [1] https://simonwillison.net/2025/Aug/7/gpt-5/#and-some-svgs-of... [2] https://www.lesswrong.com/posts/xwdRzJxyqFqgXTWbH/how-does-a... | | |
| ▲ | SR2Z a day ago | parent | next [-] | | The "pelican on a bicycle" test has been around for six months and has been discussed a ton on the internet; that second example is fascinating but Wikipedia has infoboxes containing coordinates like 48°51′24″N 2°21′8″E (Paris, notoriously on land). How much would you bet that there isn't a CSV somewhere in the training set exactly containing this data for use in some GIS system? I think that "modeling the world" is a red herring, and that fundamentally an LLM can only model its input modalities. Yes, you could say this about human beings, but I think a more useful definition of "model the world" is that a model needs to realize any facts that would be obvious to a person. The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model. | | |
| ▲ | Terr_ a day ago | parent | next [-] | | > Wikipedia has infoboxes containing coordinates like 48°51′24″N 2°21′8″E I imagine simply making a semitransparent green land-splat in any such Wikipedia coordinate reference would get you pretty close to a world map, given how so much of the ocean won't get any coordinates at all... Unless perhaps the training includes a compendium of deep-sea ridges and other features. | |
| ▲ | skissane a day ago | parent | prev | next [-] | | > The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model. A lot of humans contradict themselves all the time… therefore they cannot have any kind of sophisticated world model? | | |
| ▲ | SR2Z 12 hours ago | parent [-] | | A human generally does not contradict themselves in a single conversation, and if they do they generally can provide a satisfying explanation for how to resolve the contradiction. |
| |
| ▲ | hackinthebochs a day ago | parent | prev [-] | | >How much would you bet that there isn't a CSV somewhere in the training set exactly containing this data for use in some GIS system? Maybe, but then I would expect more equal performance across model sizes. Besides, ingesting the data and being able to reproduce it accurately in a different modality is still an example of modeling. It's one thing to ingest a set of coordinates in a CSV indicating geographic boundaries and accurately reproduce that CSV. It's another thing to accurately indicate arbitrary points as being within the boundary or without in an entirely different context. This suggests a latent representation independent of the input tokens. >I think that "modeling the world" is a red herring, and that fundamentally an LLM can only model its input modalities. There are good reasons to think this isn't the case. To effectively reproduce text that is about some structure, you need a model of that structure. A strong learning algorithm should in principle learn the underlying structure represented with the input modality independent of the structure of the modality itself. There are examples of this in humans and animals, e.g. [1][2][3] >I think a more useful definition of "model the world" is that a model needs to realize any facts that would be obvious to a person. Seems reasonable enough, but it is at risk of being too human-centric. So much of our cognitive machinery is suited for helping us navigate and actively engage the world. But intelligence need not be dependent on the ability to engage the world. Features of the world that are obvious to us need not be obvious to an AGI that never had surviving predators or locating food in its evolutionary past. This is why I find the ARC-AGI tasks off target. They're interesting, and it will say something important about these systems when they can solve them easily. But these tasks do not represent intelligence in the sense that we care about. >The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model. This proves that an LLM does not operate with a single world model. But this shouldn't be surprising. LLMs are unusual beasts in the sense that the capabilities you get largely depend on how you prompt it. There is no single entity or persona operating within the LLM. It's more of a persona-builder. What model that persona engages with is largely down to how it segmented the training data for the purposes of maximizing its ability to accurately model the various personas represented in human text. The lack of consistency is inherent to its design. [1] https://news.wisc.edu/a-taste-of-vision-device-translates-fr... [2] https://www.psychologicalscience.org/observer/using-sound-to... [3] https://www.nature.com/articles/s41467-025-59342-9 |
| |
| ▲ | homarp a day ago | parent | prev [-] | | and we can say that a bastardized version of the Sapir-Worf hypothesis applies: what's in the training set shapes or limits LLM's view of the world | | |
| ▲ | moron4hire a day ago | parent [-] | | Neither Sapir nor Whorf presented Linguistic Relativism as their own hypothesis and they never published together. The concept, if it exists at all, is a very weak effect, considering it doesn't reliably replicate. | | |
| ▲ | homarp a day ago | parent [-] | | i agree that's the pop name. Don't you think it replicates well for LLM though? |
|
|
| |
| ▲ | ajross a day ago | parent | prev [-] | | > Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise What does "understanding a sunrise" mean though? Arguments like this end up resting on semantics or tautology, 100% of the time. Arguments of the form "what AI is really doing" likewise fail because we don't know what real brains are "really" doing either. I mean, if we knew how to model human language/reasoning/whatever we'd just do that. We don't, and we can't. The AI boosters are betting that whatever it is (that we don't understand!) is an emergent property of enough compute power and that all we need to do is keep cranking the data center construction engine. The AI pessimists, you among them, are mostly just arguing from ludditism: "this can't possibly work because I don't understand how it can". Who the hell knows, basically. We're at an interesting moment where technology and the theory behind it are hitting the wall at the same time. That's really rare[1], generally you know how something works and applying it just a question of figuring out how to build a machine. [1] Another example might be some of the chemistry fumbling going on at the start of the industrial revolution. We knew how to smelt and cast metals at crazy scales well before we knew what was actually happening. Stuff like that. | | |
| ▲ | subjectivationx 18 hours ago | parent | next [-] | | Everyone reading this understands the meaning of a sunrise. It is a wonderful example of the use theory of meaning. If you raised a baby inside a windowless solitary confinement cell for 20 years and then one day show them the sunrise on a video monitor, they still don't understand the meaning of a sunrise. Trying to extract the meaning of a sunrise by a machine from the syntax of a sunrise data corpus is just totally absurd. You could extract some statistical regularity from the pixel data of the sunrise video monitor or sunrise data corpus. That model may provide some useful results that can then be used in the lived world. Pretending the model understands a sunrise though is just nonsense. Showing the sunrise statistical model has some use in the lived world as proof the model understands a sunrise I would say borders on intellectual fraud considering a human doing the same thing wouldn't understand a sunrise either. | | |
| ▲ | ajross 17 hours ago | parent [-] | | > Everyone reading this understands the meaning of a sunrise For a definition of "understands" that resists rigor and repeatability, sure. This is what I meant by reducing it to a semantic argument. You're just saying that AI is impossible. That doesn't constitute evidence for your position. Your opponents in the argument who feel AGI is imminent are likewise just handwaving. To wit: none of you people have any idea what you're talking about. No one does. So take off the high hat and stop pretending you do. | | |
| ▲ | meroes 14 hours ago | parent [-] | | This all just boils down to the Chinese Room thought experiment, where Im pretty sure the consensus is nothing in the experiment (not the person inside, the whole emergent room, etc) understands Chinese like us. Another example by Searle is a computer simulating digestion is not digesting like a stomach. The people saying AI can’t form from LLMs are in the consensus side of the Chinese Room. The digestion simulator could tell us where every single atom is of a stomach digesting a meal, and it’s still not digestion. Only once the computer simulation breaks down food particles chemically and physically is it digestion. Only once an LLM received photons or has a physical capacity to receive photons is there anything like “seeing a night sky”. |
|
| |
| ▲ | a day ago | parent | prev | next [-] | | [deleted] | |
| ▲ | pastel8739 a day ago | parent | prev [-] | | Is it really so rare? I feel like I know of tons of fields where we have methods that work empirically but don’t understand all the theory. I’d actually argue that we don’t know what’s “actually” happening _ever_, but only have built enough understanding to do useful things. | | |
| ▲ | ajross a day ago | parent [-] | | I mean, most big changes in the tech base don't have that characteristic. Semiconductors require only 1920's physics to describe (and a ton of experimentation to figure out how to manufacture). The motor revolution of the early 1900's was all built on well-settled thermodynamics (chemistry lagged a bit, but you don't need a lot of chemical theory to burn stuff). Maxwell's electrodynamics explained all of industrial electrification but predated it by 50 years, etc... | | |
| ▲ | skydhash a day ago | parent [-] | | Those big changes always happens because someone presented a simpler model that explains stuff enough we can build stuff on it. It's not like semiconductors raw materials wasn't around. The technologies around LLMs is fairly simple. What is not is the actual size of data being ingested and the number of resulting factors (weight). We have a formula and the parameters to generate grammatically perfect text, but to obtain it, you need TBs of data to get GBs of numbers. In contrast something like TM or Church's notation is pure genius. Less than a 100 pages of theorems that are one of the main pillars of the tech world. | | |
| ▲ | ajross 17 hours ago | parent [-] | | > Those big changes always happens because someone presented a simpler model that explains stuff enough we can build stuff on it. Again, no it doesn't. It didn't with industrial steelmaking, which was ad hoc and lucky. It isn't with AI, which no one actually understands. | | |
| ▲ | skydhash 12 hours ago | parent [-] | | I’m pretty sure there were always formula for getting high quality steel even before the industrial age. And you only need a few textbooks and papers to understand AI. |
|
|
|
|
|
|
|
| ▲ | jhanschoo a day ago | parent | prev | next [-] |
| Let's make this more concrete than talking about "understanding knowledge". Oftentimes I want to know something that cannot feasibly be arrived at by reasoning, only empirically. Remaining within the language domain, LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized. Similarly, you need a robot that can interact with the world and reason from newly collected empirical data in order to answer these empirical questions, if the work had not already been done previously. |
| |
| ▲ | skydhash a day ago | parent | next [-] | | > LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized But their usefulness is only surface-deep. The news that matters to you is always deeply contextual, it's not only things labelled as breaking news or happening near you. Same thing happens with code organization. The reason is more human nature (how we think and learn) than machine optimization (the compiler usually don't care). | |
| ▲ | awesome_dude a day ago | parent | prev [-] | | I know the attributes of an Apple, i know the attributes of a Pear. As does a computer. But only i can bite into one and know without any doubt what it is and how it feels emotionally. | | |
| ▲ | scrubs a day ago | parent | next [-] | | You have half a point. "Without any doubt" is merely the apex of a huge undefined iceberg. I write half .. eating is multi modal and consequential. The llm can read the menu, but it didn't eat the meal. Even humans are bounded. Feeling, licking, smelling, or eating the menu still is not eating the meal. There is an insuperable gap in the analogy ... a gap in the concept and of sensory data doing it. Back to first point: what one knows through that sensory data ... is not clear at present or even possible with llms. | | | |
| ▲ | zaphirplane a day ago | parent | prev [-] | | We segued to conscience and individuality. |
|
|
|
| ▲ | vlovich123 a day ago | parent | prev [-] |
| If it was modeling the world you’d expect “give me a picture of a glass filled to the brim” to actually do that. It’s inability to correctly and accurately combine concepts indicates it’s probably not building a model of the real world. |
| |
| ▲ | p1esk a day ago | parent [-] | | I just gave chatgpt this prompt - it produced a picture of a glass filled to the brim with water. | | |
| ▲ | jdiff a day ago | parent [-] | | Like most quirks that spread widely, a bandaid is swiftly applied. This is also why they now know how many r's are in "strawberry." But we don't get any closer to useful general intelligence by cobbling together thousands of hasty patches. | | |
| ▲ | llbbdd 19 hours ago | parent [-] | | Seems to have worked fine for humans so far. | | |
| ▲ | bigstrat2003 14 hours ago | parent [-] | | No, humans are not a series of band-aid patches where we learn facts in isolation. A human can reason, and when exposed to novel situations figure out a path forward. You don't need to tell a human how many rs are in "strawberry"; as long as they know what the letter r is they can count it in any word you choose to give them. As proven time and time again, LLMs can't do this. The embarrassing failure of Claude to figure out how to play Pokemon a year or so ago is a good example. You could hand a five year old human a Gameboy with Pokemon in it, and he could figure out how to move around and do the basics. He wouldn't be very good, but he would figure it out as he goes. Claude couldn't figure out to stop going in and out of a building. LLMs, usefulness aside, have repeatedly shown themselves to have zero intelligence. | | |
| ▲ | llbbdd 13 hours ago | parent [-] | | I was referring not to individual learning ability but to natural selection and evolutionary pressure, which IMO is easy to describe as a band-aid patch that takes a generation or more to apply. | | |
| ▲ | vlovich123 12 hours ago | parent [-] | | You would be correct if these issues were fixed by structurally fixing the LLM. But instead it’s patched through RL/data set management. That’s a very different and more brittle process - the evolutionary approach fixes classes of issues while the RL approach fixes specific instances of issues. | | |
| ▲ | llbbdd 8 hours ago | parent [-] | | Sure, and I'd be the first to admit I'm not aware of the intricate details wrt how LLMs are trained and refined, it's not my area. My original comment here was in disagreement of the relatively simple dismissal of the idea that the construction of humanity hasn't been an incremental zig-zag process and that I don't see any reason that a "real" intelligence couldn't follow the same path under our direction. I see a lot of philosophical conversation around this on HN disguised as endless deep discussions about the technicals, which amuses me because it feels like we're in the very early days there, and I think we can circle the drain defining intelligence until we all die. |
|
|
|
|
|
|
|
