| ▲ | rfgplk 4 hours ago | |||||||
What's the perplexity of the average person? | ||||||||
| ▲ | kingstnap 3 hours ago | parent | next [-] | |||||||
Beating even a very small LLM at next token prediction is basically impossible. [2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2 We knew this many years ago. Like GPT 1 era. Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before. Language modelling game https://rr-lm-game.herokuapp.com/ | ||||||||
| ||||||||
| ▲ | jrflo 3 hours ago | parent | prev | next [-] | |||||||
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain... | ||||||||
| ||||||||
| ▲ | Kotlopou 3 hours ago | parent | prev | next [-] | |||||||
I linked a forum post asking this exact question elsethread (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...), and the answer is frustratingly dependent on the context length, the dataset, and the knowledge of the person about the topic of that dataset (e.g. I would have high perplexity in American history). | ||||||||
| ▲ | 3 hours ago | parent | prev [-] | |||||||
| [deleted] | ||||||||