Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.

▲

sigbottle 2 hours ago | parent | next [-]

If you want to take it that far, we've had results like this since the 60s (solomonoff induction). But of course if you state it like that, your (rightful) objection is that it's pretty vacuous (if I had the computational omniscience to just brute force all possible turing machines, whatever that even means, then sure, any 'f' gets subsumed into this paradigm).

A lot of philosophers, mathematicians, scientists, etc. effectively say, "Yeah, everything's just f(inputs of world) -> outputs!" That 'f' is doing a lot of heavy lifting. Which is kind of the point of mechanistic interperability - to make sure we're not jumping ahead of ourselves, and to make sure we're careful when we claim what "deep" and "structure" means, when it pertains to that 'f'.

▲

guhcampos 4 hours ago | parent | prev | next [-]

Isn't that what "Attention is all you need" was about anyway? Does not sound like news to me.

▲

bunderbunder 3 hours ago | parent | next [-]

It possibly goes a little further, though, since we’re talking about multi-step agentic loops here. So in addition to next token prediction there are of iterations of collecting input from outside sources.

I’m still hesitant to interpret this as “thinking ahead” without at least seeing some more back-and-forth in the literature first, though. This just seems like one of those spots where it makes sense to give other researchers some time to come up with additional hypotheses to explain the observations instead of focusing on the first one anyone proposes in isolation.

▲

FeepingCreature 3 hours ago | parent | prev [-]

This is all obvious and not news, but there's a lot of people doing fighting retreats against LLM intelligence where the degree of obviousness matters.

	▲	XenophileJKO 31 minutes ago \| parent [-]
		Anticipatory token selection was obvious when you used structured output on gpt 3.5. The model would construct/hallucinate pre-conditions to satisfy some final output value that the model was predisposed to.

▲

chrisjj 4 hours ago | parent | prev | next [-]

> To effectively predict the next token it needs a good idea of what comes after the next token.

And that's all it needs. Not reasoning.

▲

brookst 3 hours ago | parent | next [-]

Save us from the reasoning / sentience / consciousness / thinking semantic quicksand.

Babbage’s Analytical Engine didn’t actually analyze anything, and terminology hadn’t gotten any more clear-cut since.

	▲	RetroTechie an hour ago \| parent [-]
		+1. Inputs, processing & memory do the job. I suspect exact and/or universal definitions for intelligence, self-awareness, 'feelings' etc will prove to be elusive, and best we'll get is systems/robots etc behaving as if possessing those qualities. With some tests to put a number on them. Downside is that may apply to us humans too.

▲

wongarsu 3 hours ago | parent | prev | next [-]

At some level of performance reasoning becomes the most effective method to predict the next token

▲

chrisjj 3 hours ago | parent [-]

Chatbot "reasoning" arises from some level of the performance of the particular user, not of the chatbot.

	▲	astro1234 2 hours ago \| parent [-]
		I’m curious what this means? I think the evidence is pretty convincing that, while brittle, there is reasoning going on (though it depends on your definition of reasoning which I’m curious what that is for you).

▲

rlt 2 hours ago | parent | prev | next [-]

Isn't "reasoning" in LLMs just training it to have an internal monologue to think through problems like a human would? i.e. extra tokens.

▲

vidarh 3 hours ago | parent | prev | next [-]

How do you define reasoning in a measurable way?

▲

anuramat 3 hours ago | parent [-]

I keep asking the same question, and I think the steelman version would be "has metacognitive patterns similar to humans"

▲

pixl97 2 hours ago | parent [-]

I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all. This just changes the argument from LLMs can reason like a human to LLMs can't reason at all while ignoring the third possibility of "LLMs can reason not like a human".

One of the biggest things I've learned after the event of LLMs is that humans definitions of intelligence/thinking/reasoning/consciousness/etc are very poorly defined. Not just across society at large, but the sciences themselves.

	▲	sigbottle an hour ago \| parent \| next [-]
		> I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all. Something like this is actually a stance in the tradition of inferentialism (see the term sapience). Though "reasoning" isn't like, turing machine computability in this space; from what I understand, it's some abstract notion of the "space of reasons". I don't really understand it, honestly. There's some merit to this, IMO. When an LLM goes wrong, do you blame the person or the LLM? As in, would you throw said LLM in jail, and hold the LLM accountable? Not right now, at least. I'm not sure if that's what is meant by the "space of reasons", but the intuition is that 'reason' can mean a lot of different things, pragmatically speaking. Reason as a legible audit trail is one of those ways. But that's arguably getting into the social aspect of 'reason' (important!) and not like, what STEM people traditionally think of as 'reason'.
	▲	AlienRobot 23 minutes ago \| parent \| prev [-]
		I may be wrong but I think the word you want is anthropocentric, not anthropomorphic. Anthropomorphism is turning something that isn't human into something human, e.g. an AI, while anthropocentrism is taking human as the center and default state of things.

▲

jimbokun 3 hours ago | parent | prev [-]

What’s the difference?

	▲	chrisjj 40 minutes ago \| parent [-]
		Is regexp or compiler lookahead reasoning? No. Mistaking chatbot lookahead as reasoning comes with being gulled by the "artificial intelligence" sales pitch.

▲

IAmGraydon 4 hours ago | parent | prev [-]

I think that's reading a little too much into it. The paper shows the hidden states contain signals about whether the code is good right now, and whether the run is probably going to work out. That's interesting, for sure. But it doesn't mean the model has some detailed idea of what it's going to write 25 steps later.

A lot of that signal could be much simpler stuff. This task is hard. The agent seems stuck. The tests are getting better. The current approach looks promising. All of those things make future success easier to predict without the model actually "knowing what comes next" in any strong sense.

Also, their 25 steps are agent turns, not 25 code edits. The median run had something like 52 steps but only two edits, and the program label stays the same between edits. So "25 steps ahead" may sometimes just mean basically the same codebase, with a bunch of reading and test output in between.

So yeah, I'd say it's consistent with Sutskever's view. But "consistent with" and "confirmatory of" are doing very different amounts of work here.

	▲	ahk-dev 2 hours ago \| parent [-]
		I think that's the practical implication. Whether it's "thinking ahead" or not, if the model can reliably predict a failing trajectory several agent turns in advance, an agent could backtrack or branch much earlier instead of spending tokens on a dead end. That seems more interesting than the terminology debate.