I made a general purpose harness integrated into MelonDS and got Claude to play Mario Kart by feeding it continuous video.

It made forward progress in the Figure 8 circuit after I helped it through a menu but kept slamming into a wall so it wasn't on track to win in less than an hour.

Also got it to play Age of Empires: Age of Kings using the same technique but it failed to click on anything.

DS specifically is very fun because it's touch based but the UI components aren't accessible. So it is extremely challenging for LLM's spatial reasoning skills.

I want to improve the harness more and have the LLM dynamically create its own tools based on drawing grid box overlays on a screen in a feedback loop, so it can say "click on the 'end turn'" button instead of "click 240,320" and it would 'just work' in any game.

I also want to eventually play games with it... I didn't really have friends to play my massive DS library with as a kid so it'd be nice to finally have someone that can roast me or react to my skills. And learn my playstyle enough to punish me.

Unfortunately haven't had the time due to work at my day job and needing to clean out my apartment.

If you get the opportunity, I really want to know how Claude does with Mario Party.

I also am personally curious how the GPT models (which advertise better computer use, etc.) would do as compared to Claude.