One limitation of these type of demos (and similar with Minecraft, etc) is that: 1. Putting into "peaceful" mode makes response time a non-factor 2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game.

This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.

At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).

Isn’t the point of arc agi 3 to measure general game intelligence even if it’s not a AAA game? Frontier models have obliterated that. And they do keep most of it private so no gaming there.

The models these days have 3 components that is for sure going to get better and better

1. Very good vision (it can identify almost anything in a picture) 2. Keyboard and mouse actions (slow but it does it well) 3. General world understanding (this one is toss up but given Astra or Fable, it surely does well even if it takes a while)

Now combine all these with specific RL environments that may use specific games which also generalizes to an extent and we have a system that can only get significantly better from here.

As for speed, I’d say it’s a question of how much money you are willing to throw (like Cerebras for example).

Good point. There are also a huge amount of open source calculators, planners and other tools that these models have undoubtably been trained on [0]

There are full blueprints for factories online available to copy-paste in its training data [1]

I wouldn't be surprised if you can beat the game by just copying a bunch of blueprints from the web and chaining them together.

This is an old game relatively speaking and its been optimized to hell.

0: https://factoriolab.github.io/2.1

1: https://factorioprints.com

These are really two different challenges - playing the game for the first time with exploration, and beating the game with all the prior knowledge.

Shhh you can't say anything anti-hype here.