Remix.run Logo
▲ chubot 3 hours ago

I thought that too, but models are trained on observation now, to increase agent performance. This is new data that is separate from Internet crawls. I think it's done with reinforcement learning with verifiable rewards

https://github.com/opendilab/awesome-RLVR

Examples:

- When playing Go or chess, nobody may have written or published web pages with a particularly clever end game, but an agent can use CPU time to discover these games themselves, and learn from them. Human Go players report learning strategy from what AlphaGo discovered.

- Agents use Unix shell and are trained with it. So they can discover things and combine things that nobody has written about on the Internet, like how to debug a rarely used driver, or an interaction between two drivers. And they can see patterns in all the drivers that humans may have a hard time even seeing, leaving aside the motivation to publish on the Internet.

This explains why the performance is so "jagged" - great on coding, math and closed games, but less good on things involving the real world. We probably did hit a data wall, but then we got new sources of data.

▲rhdunn 2 hours ago | parent [-]

An interesting experiment would be to create a transformer model that is just designed to play Chess or Go. Specifically, the only thing that should change is the number of tokens, consisting of:

- the row of the move;

- the column of the move;

- the piece moved;

- any special/additional states from that move -- check, checkmate, castle, etc.;

- maybe an indication of whose turn it is (i.e. a black piece and a white piece token).

You could then train the base model on published game transcripts and use the interactive RL approach like AlphaZero and LeelaChess use, as well as what you mention for LLMs.

Why this is interesting is that it should give insights into how LLMs/transformers learn Chess/Go without having other data affecting the results. Specifically whether the token semantics identify valid moves, etc. and whether the context window encodes gameplay strategies across the different layers.