Shouldn't self-driving be a piece of cake if it works this well for Doom? Or what am I missing?

The model doesn't have image input capabilities (yet, it seems from the post), so for the Doom demo, a harness is extracting a bunch of structured information from the game (map layout, enemy locations, player ammo, health, etc) and providing it as a massive JSON blob to the model so it can make its decisions. This model _could_ be hooked up to make the decisions for a self-driving car, but it would need to be fed a structured blob of the situation around it, so all the computer vision problems of self-driving are still there. And that's before you get into the confidence and accuracy of this model.

I think the doom demo uses a text representation of the world and it's basically, "projectile coming your way" -> "Strafe". "Enemy ahead" -> "shoot. So it works well when spawned in a room of enemies (as we see in the video).

If self driving is red means stop, green means go, and stay in your lane - then it would work great, but having to actually think and test which maneuver is optimal for a given situation while weighting safety, road rules, random unexpected actions and getting to your destination, I think it's a much bigger problem. A bigger model specifically trained on that maybe would do great, but then the output is not the constraint anymore.

But I haven't tried the model, so I 'm just ballparking and could be very wrong.

Driving is more complicated than Doom, and it doesn't look that great at Doom to me.