Remix.run Logo
▲ kingstnap 3 hours ago

Beating even a very small LLM at next token prediction is basically impossible.

[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2

We knew this many years ago. Like GPT 1 era.

Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.

Language modelling game https://rr-lm-game.herokuapp.com/

▲rfgplk 3 hours ago | parent [-]

Will try this out, thanks