Ok! will give it a shot. In a few iterations I gave him screenshots, i have given him the ability to take screenshots, and I gave him the Playwright MCP. I kind of gave up on the path you're suggesting (though I didn't get super far along) because I felt like I would run into this problem eventually of needing a model to figure out what a planet is, where the edge of the planet is, etc.

But if that could be done deterministically, I totally agree this is the way to go. I'll put some more time into it over the next couple weeks.