Pitching in on this with a tangent - how good are LLMs with RTS games these days? As someone without friends into that genre, it’d be pretty cool to play eg. AoE II against a capable computer that play like a real human…

Depends on what you mean, LLMs can probably _make_ pretty good AIs. It'll have all the AI scripts in the base game, including the three iterations (base, FE, DE) all the user generated ones ( including barbarian ) and then able to consume the language schema. Rig up a baby model that takes the matchup during loading and hot swaps one of your pregenerated AI scripts.

If you meant _playing_ raw based on LLM input - that's probably the wrong tool for the job. The latency for you to react to a mango shot is faster than a billion tok/s lol

Instead of driving the agent with an llm, it might work to use the agent to hard code heuristics, and use some kind of a simulation to benchmark its skills? Then feeding the results back to the agent so it can improve the heuristics?

It's improving but sota models are now too slow for a real time game. Training a specialized neural network would be more effecient.