Remix.run Logo
▲ eulgro 3 hours ago

Didn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?

I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?

▲chubot 3 hours ago | parent | next [-]

I thought that too, but models are trained on observation now, to increase agent performance. This is new data that is separate from Internet crawls. I think it's done with reinforcement learning with verifiable rewards

https://github.com/opendilab/awesome-RLVR

Examples:

- When playing Go or chess, nobody may have written or published web pages with a particularly clever end game, but an agent can use CPU time to discover these games themselves, and learn from them. Human Go players report learning strategy from what AlphaGo discovered.

- Agents use Unix shell and are trained with it. So they can discover things and combine things that nobody has written about on the Internet, like how to debug a rarely used driver, or an interaction between two drivers. And they can see patterns in all the drivers that humans may have a hard time even seeing, leaving aside the motivation to publish on the Internet.

This explains why the performance is so "jagged" - great on coding, math and closed games, but less good on things involving the real world. We probably did hit a data wall, but then we got new sources of data.

▲rhdunn 2 hours ago | parent [-]

An interesting experiment would be to create a transformer model that is just designed to play Chess or Go. Specifically, the only thing that should change is the number of tokens, consisting of:

- the row of the move;

- the column of the move;

- the piece moved;

- any special/additional states from that move -- check, checkmate, castle, etc.;

- maybe an indication of whose turn it is (i.e. a black piece and a white piece token).

You could then train the base model on published game transcripts and use the interactive RL approach like AlphaZero and LeelaChess use, as well as what you mention for LLMs.

Why this is interesting is that it should give insights into how LLMs/transformers learn Chess/Go without having other data affecting the results. Specifically whether the token semantics identify valid moves, etc. and whether the context window encodes gameplay strategies across the different layers.

▲jrflo 3 hours ago | parent | prev | next [-]

Yes, they started to hit a data wall, but I thought that much of the progress of the last couple years came from reinforcement learning after pretraining, which is mostly building complex environments for the models to reason in, rather than accumulating piles of raw text.

▲redox99 2 hours ago | parent | prev [-]

I'd hope they measured PPL on the same dataset for all those models. And one that's fair and doesn't benefit some models more than others (probably non synthetic dataset).