The middle ground would be leveraging e. g. standard a11y APIs, and/or hooking into applications like Squish does.

Then you get a nice textual world that fits the LLM without having to rewrite every application to have a fullblown HTTP server.