This is why I'm very skeptical about the "Nobel prize level" claims. To win a Nobel prize you would have to produce something completely new. LLM will probably be able to reach a Ph.D. level of understanding existing research, but bringing something new is a different matter.

▲

adamzwasserman 3 days ago | parent | next [-]

LLMs do not understand anything.

They have a very complex multidimensional "probability table" (more correctly a compressed geometric representation of token relationships) that they use to string together tokens (which have no semantic meaning), which then get converted to words that have semantic meaning to US, but not to the machine.

▲

DoctorOetker 3 days ago | parent | next [-]

Consider your human brain, and the full physical state, all the protons and neutrons some housed together in the same nucleus, some separate, together with all the electrons. Physics assigns probabilities to future states. Suppose you were in the middle of a conversation and about to express a next syllable (or token). That choice will depend on other choices ("what should I add next"), and further choices ("what is the best choice of words to express the thing I chose to express next etc. The probabilities are in principle calculable given a sufficiently detailed state. You are correct that LLM's correspond to a probability distribution (given you immediately corrected to say that this table is implicit and parametrized by a geometric token relationships.). But so does every expressor of language, humans included.

The presence or absence of understanding can't be proven by mere association of with a "probability table", especially if such probability table is exactly expected from the perspective of physics, and if the models have continuously gained better and better performance by training them directly on human expressions!

▲

tomfly 3 days ago | parent | prev | next [-]

Exactly. It’s been stated for a long time, before llms. For instance this paper https://home.csulb.edu/~cwallis/382/readings/482/searle.mind... Describes a translator who doesn’t know the language.

▲

KoolKat23 3 days ago | parent | prev [-]

In abstract we do the exact same thing

▲

adamzwasserman 3 days ago | parent | next [-]

Perhaps in practice as well. It is well-established that our interaction with language far exceeds what we are conscious of.

	▲	KoolKat23 3 days ago \| parent [-]
		Absolutely, it is world model building.

▲

tomfly 3 days ago | parent | prev [-]

It’s hard to believe this when the llm “knows” so much more then us yet still can not be creative outside its training distribution

	▲	KoolKat23 3 days ago \| parent \| next [-]
		When are we as humans creative outside our training data? It's very rare we actually discover something truly novel. This is often random, us stumbling onto it, brute force or purely by being at the right place at the right time. On the other hand, until it's proven it'd likely be considered a hallucination. You need to test something before you can dismiss it. (They did burn witches for discoveries back in the day, deemed witchcraft). We also reduce randomness and pre-train to avoid overfitting. Day to day human creative outputs as humans are actually less exciting when you think about it further, we build on pre-existing knowledge. No different to good prompt output with the right input. Humans are just more knowledgeable & smarter at the moment.
	▲	adamzwasserman 3 days ago \| parent \| prev [-]
		The LLM doesn't 'know' more than us - it has compressed more patterns from text than any human could process. That's not the same as knowledge. And yes, the training algorithms deliberately skew the distribution to maintain coherent output - without that bias toward seen patterns, it would generate nonsense. That's precisely why it can't be creative outside its training distribution: the architecture is designed to prevent novel combinations that deviate too far from learned patterns. Coherence and genuine creativity are in tension here

▲

KoolKat23 3 days ago | parent | prev [-]

Given a random prompt, the overall probability of seeing a specific output string is almost zero, since there are astronomically many possible token sequences.

The same goes for humans. Most awards are built on novel research built on pre-existing works. This a LLM is capable of doing.

▲

adamzwasserman 3 days ago | parent [-]

LLMs don't use 'overall probability' in any meaningful sense. During training, gradient descent creates highly concentrated 'gravity wells' of correlated token relationships - the probability distribution is extremely non-uniform, heavily weighted toward patterns seen in training data. The model isn't selecting from 'astronomically many possible sequences' with equal probability; it's navigating pre-carved channels in high-dimensional space. That's fundamentally different from novel discovery.

▲

KoolKat23 3 days ago | parent [-]

That's exactly the same for humans in the real world.

You're focusing too close, abstract up a level. Your point relates to the "micro" system functioning, not the wider "macro" result (think emergent capabilities).

▲

adamzwasserman 3 days ago | parent [-]

I'm afraid I'd need to see evidence before accepting that humans navigate 'pre-carved channels' in the same way LLMs do. Human learning involves direct interaction with physical reality, not just pattern matching on symbolic representations. Show me the equivalence or concede the point.

▲

KoolKat23 3 days ago | parent [-]

Language and math are a world model of physical reality. You could not read a book and make sense of it if this were not true.

An apple falls to the ground because of? gravity.

In real life this is the answer, I'm very sure the pre-carved channel will also lead to gravity.

▲

adamzwasserman 3 days ago | parent [-]

You're proving my point. You know the word 'gravity' appears in texts about falling apples. An LLM knows that too. But neither you nor the LLM discovered gravity by observing reality and creating new models. You both inherited a pre-existing linguistic map. That's my entire argument about why LLMs can't do Nobel Prize-level work.

▲

KoolKat23 2 days ago | parent [-]

Well it depends. It doesn't have arms and legs so can't physically experiment in the real world, a human is currently a proxy for that, we can do it's bidding and feedback results though, so it's not really an issue.

Most of the time that data is already available to it and they merely need to a prove a thereom using existing historic data points and math.

For instance the Black-Scholes-Merton equation which won the Nobel economics prize was derived using preexisting mathematical concepts and mathematical principles. The application and validation relied on existing data.

▲

adamzwasserman 2 days ago | parent [-]

The Black-Scholes-Merton equation wasn't derived by rearranging words about financial markets. It required understanding what options are (financial reality), recognizing a mathematical analogy to heat diffusion (physical reality), and validating the model against actual market behavior (empirical reality). At every step, the discoverers had to verify their linguistic/mathematical model against the territory.

LLMs only rearrange descriptions of discoveries. They can't recognize when their model contradicts reality because they never touch reality. That's not a solvable limitation. It's definitional.

We're clearly operating from different premises about what constitutes discovery versus recombination. I've made my case; you're welcome to the last word

	▲	KoolKat23 2 days ago \| parent [-]
		I understand your viewpoint. LLM's these days have reasoning and can learn in context. They do touch reality, your feedback. It's also proven mathematically. Other people's scientific papers are critiqued and corrected as new feedback arrives. This is no different to claude code bash testing and fixing it's own output errors recursively until the code works. They already deal with unknown combinations all day, our prompting. Yes it is brittle though. They are also not very intelligent yet.