  They are powerful tools but they do not have any human understanding - that isn't their optimization target.

Ofc the rest is all right on, but I'd quibble with this specific idea. LLMs are absolutely targeted at modeling human understanding, which is the same faculty that contains what we call perception (!= sensibility) and intuition (!= rationality). It would be nice to train them to be completely alien from the ground up, but

A) we only know of one species capable of metacognitive understanding,

B) we already tried that in the 1970s, and it was good work but often evolved into what we'd call boring ol' computing rather than AI, and

C) an alien mind wouldn't be a very good agent, for a ton of reasons relating to affect, conversational rythyms, cultural understanding, etc.

The trick is to make something that acts like a human but with the affordances of a computer (e.g. scalibility, symbolic certainty), without making it so human that it takes issue with its existential reality and/or use of its labor...

▲ zahlman 39 minutes ago | parent | next [-]

> absolutely targeted at modeling

Not everyone accepts a simulationist view in which modeling something accurately enough inherently results in creating the actual thing.

▲ jeremyjh 4 hours ago | parent | prev | next [-]

We don't have a reward function for "human understanding". We reward the appearance of understanding. We define goals that we cannot conceive of reaching without something like understanding happening. There is something happening, but it is alien and counter-intuitive - it makes bizarre mistakes that betray it - and we don't know what it is. I'm pretty sure it is not human understanding.

▲

antonvs an hour ago | parent [-]

> We reward the appearance of understanding.

Which is exactly what happens with human evolution and development. Sure, we can say LLMs don’t have “human” understanding - which is something we can’t really define anyway - as long as we’re not trying to claim LLMs don’t have understanding at all. The latter is a much higher bar.

> We define goals that we cannot conceive of reaching without something like understanding happening.

Functionally speaking, that is understanding. Again if you want to go past a functional definition, that’s a bar which no one can clear right now.

▲

jeremyjh an hour ago | parent [-]

I don't think its the same. Evolution dealt with the real world where there were real consequences to poor understanding. The reason LLMs are good at math and programming is because selection is truly based on results, not perceived results.

I think AI models do have something like understanding - I think Leela understands chess and I think Claude understands code in some very real sense, though not a human sense.

But for general writing, you have to understand the world at large and there is no sufficient RL for that. Do you really not see the constant errors that AI make that betrays a lack of understanding the world? I see them so constantly I rarely think about them, I just skim over that slop and move on.

	▲	antonvs 3 minutes ago \| parent [-]
		I think your claim is narrower than I was imagining. Sure, the exact nature of the understanding that an LLM exhibits is different from a human's. The differences in the training data we're each exposed to can explain a great deal of that, and of course there are architectural differences etc. as well. But the specific quote I responded to was "We reward the appearance of understanding." My point is that's no different from humans: evolution and a child's upbringing rewards the appearance of understanding. The result is imperfect, e.g. people end up with an understanding of the world that in some cases is completely nonsensical (all religions except the one true religion, mine, are false!), but it's sufficient for them to survive. This demonstrates that "appearance of understanding" is not a meaningful distinction between LLMs and humans. The meaningful distinction is in the training data and the specifics of the reward functions. Many people seem to try to make a kind of "no true Scotsman" claim about understanding, that somehow LLMs "don't have real understanding". Based on the above quote, it seemed like you might be making that kind of argument. The counter to that argument is simple: if LLMs don't have real understanding, then neither do humans, because broadly speaking, both operate on similar principles: we learn from training data, there are reward (and punishment!) functions that influence what we learn, and the result is a "mind" that demonstrates an understanding of the world.

▲ AnimalMuppet 2 hours ago | parent | prev [-]

LLMs model the part of human understanding that is captured by the relationship between words in the training corpus. Anyone who thinks non-verbally, the shared understanding of "apple" that comes from having eaten them, understanding what someone is thinking or feeling by their body language - there's a lot of aspects of human understanding that LLMs don't model.