This is sick, thanks for sharing! We've been working on very similar things for the past 2 years. We also started with a text-only representation, but sadly quickly realized that only a small subset of games work well with this.

So we went down a rabbit hole and decided to do everything purely based on pixels and OS inputs.

We're currently only live for mobile but happy to give you early access to nunu ai for PC if interested. Would love to see how we compare!