What does "predicting the next token" mean? I ask this every time people say "LLMs are just predicting the next token" and it's maddening that nobody can give a straight answer. Predicting it according to what probability distribution? Every process that produces a sequence of actions (including e.g. a human writing) can be modeled by some probability distribution and therefore their actions are indistinguishable from "predicting the next token" emitted by that distribution.

Yeah that's pretty much what gwern argues here[0]. Or to adapt another proverb: to predict the next token you first need to model the universe.

[0] https://gwern.net/scaling-hypothesis#gwern-difference--effic...