If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.

I suspect Opus 5.5 can do even better but there aren't as many people testing it.

Where are the recordings of these games? I know this kind of stuff must be happening, but it seems oddly difficult to track down the actual recordings, which feel like the really fascinating thing people should be publishing if they’re putting in the time and tokens to do stuff like this.

Like I want to see what Pokemon strategies it uses: Does it try to brute force with its starter? Does it try to load all of its HMs onto one or two Pokemon? Does it use Flash? If it needs to catch a new Pokemon to beat a gym leader, how much does it remember about which species it encountered where and what their levels were?

If you reading every wiki, forum, dev blog/patch note, watching every video, etc before playing, I’d be more surprised if you could NOT play a game.

Your last sentence mirrors my model choices recently. Opus can do things very well but Sol can do them well for less than 1/4 the price.

How do they feed the game screen to the models? I imagine the models are too slow if you send 60FPS, but I feel like that’s also the case if you maybe feed it 1FPS? If you go any slower, you’ll miss too much info?

Some of these Games are definitely in the RL environments.

Do you have link for the VII bonus boss run?