Remix.run Logo
▲ rfgplk 4 hours ago

What's the perplexity of the average person?

▲kingstnap 3 hours ago | parent | next [-]

Beating even a very small LLM at next token prediction is basically impossible.

[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2

We knew this many years ago. Like GPT 1 era.

Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.

Language modelling game https://rr-lm-game.herokuapp.com/

▲rfgplk 3 hours ago | parent [-]

Will try this out, thanks

▲jrflo 3 hours ago | parent | prev | next [-]

Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...

▲amluto 23 minutes ago | parent [-]

We are, though, but only partially. When listening and reading, we pretty clearly build a model of what we’re hearing or seeing and use it to correct for the rather noisy channels we use.

But when thinking, I’m fairly confident that our ability to predict the next few words of a book on a complex topic (in its words as opposed to the reader’s own words) is at best weakly correlated with intelligence or with subject-specific skill.

I certainly feel like I’ve learned a subject better when I can solve a problem differently from a solution I’ve seen before.

▲Kotlopou 3 hours ago | parent | prev | next [-]

I linked a forum post asking this exact question elsethread (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...), and the answer is frustratingly dependent on the context length, the dataset, and the knowledge of the person about the topic of that dataset (e.g. I would have high perplexity in American history).

▲ 3 hours ago | parent | prev [-]
[deleted]