I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us.

EDIT: gentler phrasing

▲

garrinm 4 days ago | parent | next [-]

The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.

What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.

So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.

And I think effectively the same distinction applies here.

▲

Planktonne 4 days ago | parent [-]

I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.

There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.

▲

danielmarkbruce 4 days ago | parent [-]

It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move.

So, it's not a next move predictor. It's a game result predictor.

▲

YeGoblynQueenne 3 days ago | parent [-]

Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?

▲

danielmarkbruce 3 days ago | parent [-]

Brush up :)

The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.

▲

YeGoblynQueenne 3 days ago | parent [-]

Alright, I'll have to check up on that. Thanks for being nice about it.

▲

YeGoblynQueenne 3 days ago | parent [-]

Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return.

To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode).

So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function.

Sutton & Barto 2ed is free here btw:

https://web.stanford.edu/class/psych209/Readings/SuttonBarto...

The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.

▲

danielmarkbruce 2 days ago | parent [-]

Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.

And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.

▲

YeGoblynQueenne 2 days ago | parent | next [-]

You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier).

>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.

	▲	danielmarkbruce 2 days ago \| parent [-]
		But the next move is self evident if you have a prediction of the value of being in each of the states possible.

▲

2 days ago | parent | prev [-]

[deleted]

▲

zahlman 4 days ago | parent | prev | next [-]

> generated an article that continually undermines its own main point.

I disagree that this accurately describes TFA.

▲

deathanatos 4 days ago | parent | next [-]

The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.

(I would also agree with the parent commenter on that the writing smells like an LLM.)

	▲	astrobe_ 4 days ago \| parent [-]
		The reward function seems indeed to be the protagonist there, still it stays in the shadows. One can only imagine that it is some kind of evaluator that scores the sequence based on grammar correctness, semantic consistency, etc.? To use the proposed chess analogy, maybe it could be a Stockfish engine that evaluates the submitted position that results from the move submitted by the LLM?

▲

Planktonne 4 days ago | parent | prev [-]

I'm not sure what you want me to do with that information; clearly I do think that my description is accurate.

The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.

▲

garrinm 4 days ago | parent | next [-]

It was written by a human. There are AI edits but it’s very much a human composition. Perhaps a bit sloppy.

▲

Planktonne 4 days ago | parent [-]

In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.

▲

garrinm 4 days ago | parent [-]

I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.

	▲	Planktonne 4 days ago \| parent \| next [-]
		Mostly I disagree with the article's ideas, if that helps; the AI was just a secondary factor. > I don’t think it’s a good reason to dismiss this AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years. In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI. In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.
	▲	zahlman 4 days ago \| parent \| prev [-]
		Out of curiousity, do you ask your editing system for diffs? Seems to me like the best way to notice and review whether your "voice" is degrading. Personally I would never let an LLM touch my prose (although I'd happily use it for research and paraphrase things it told me), but if I force myself to consider the idea, that seems like the first thing I'd want. Maybe upon reading a diff you'd even consider going a third way with the text.

▲

zahlman 4 days ago | parent | prev [-]

> I'm not sure what you want me to do with that information

For example, you could cite specific things that you believe to be "AI tells" or "admissions".

▲

Planktonne 4 days ago | parent | next [-]

It's a short article; you could read it. One example to get you started is the very first sentence:

> Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete.

The article is about how 'next-token predictor' is the wrong mental model; it opens with the admission that it is not the wrong mental model.

▲

zahlman 4 days ago | parent [-]

I did read it. People are allowed to disagree with your conclusions. Comment guidelines ask us all not to make such accusations.

To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").

	▲	Planktonne 4 days ago \| parent [-]
		To informally describe something as wrong in an article focused on how it's wrong to informally describe something is incoherent. There's a certain irony in pointing me towards the guidelines on the grounds that I have limited patience with your comments that violate them in various ways. I'm not sure that this is a productive discussion.

▲

angoragoats 4 days ago | parent | prev [-]

Not the person you’re replying to, but I read the whole article as an admission that it’s still a next-token predictor. More specifically: what does RLVR fundamentally change that somehow makes the whole process no longer a next-token predictor? The article makes no attempt to explain this. Additionally, I find its framing of the term “next-token predictor” as meaning “predicting the next token only based on raw training data” in common usage to be a bit dishonest.

To summarize: yes, RLVR and other synthetic training methods exist! It’s still a next-token predictor, and it does not “learn” or “think” or “reason” in the human sense, like so many people seem to believe.

▲

YeGoblynQueenne 3 days ago | parent | prev [-]

>> EDIT: gentler phrasing

Thank you.