We don't have a reward function for "human understanding". We reward the appearance of understanding. We define goals that we cannot conceive of reaching without something like understanding happening. There is something happening, but it is alien and counter-intuitive - it makes bizarre mistakes that betray it - and we don't know what it is. I'm pretty sure it is not human understanding.

▲

antonvs an hour ago | parent [-]

> We reward the appearance of understanding.

Which is exactly what happens with human evolution and development. Sure, we can say LLMs don’t have “human” understanding - which is something we can’t really define anyway - as long as we’re not trying to claim LLMs don’t have understanding at all. The latter is a much higher bar.

> We define goals that we cannot conceive of reaching without something like understanding happening.

Functionally speaking, that is understanding. Again if you want to go past a functional definition, that’s a bar which no one can clear right now.

▲

jeremyjh an hour ago | parent [-]

I don't think its the same. Evolution dealt with the real world where there were real consequences to poor understanding. The reason LLMs are good at math and programming is because selection is truly based on results, not perceived results.

I think AI models do have something like understanding - I think Leela understands chess and I think Claude understands code in some very real sense, though not a human sense.

But for general writing, you have to understand the world at large and there is no sufficient RL for that. Do you really not see the constant errors that AI make that betrays a lack of understanding the world? I see them so constantly I rarely think about them, I just skim over that slop and move on.

	▲	antonvs 3 minutes ago \| parent [-]
		I think your claim is narrower than I was imagining. Sure, the exact nature of the understanding that an LLM exhibits is different from a human's. The differences in the training data we're each exposed to can explain a great deal of that, and of course there are architectural differences etc. as well. But the specific quote I responded to was "We reward the appearance of understanding." My point is that's no different from humans: evolution and a child's upbringing rewards the appearance of understanding. The result is imperfect, e.g. people end up with an understanding of the world that in some cases is completely nonsensical (all religions except the one true religion, mine, are false!), but it's sufficient for them to survive. This demonstrates that "appearance of understanding" is not a meaningful distinction between LLMs and humans. The meaningful distinction is in the training data and the specifics of the reward functions. Many people seem to try to make a kind of "no true Scotsman" claim about understanding, that somehow LLMs "don't have real understanding". Based on the above quote, it seemed like you might be making that kind of argument. The counter to that argument is simple: if LLMs don't have real understanding, then neither do humans, because broadly speaking, both operate on similar principles: we learn from training data, there are reward (and punishment!) functions that influence what we learn, and the result is a "mind" that demonstrates an understanding of the world.