Great to see people thinking about this. But it feels like a step on the road to something simpler.

For example, web accessibility has potential as a starting point for making actions automatable, with the advantage that the automatable things are visible to humans, so are less likely to drift / break over time.

Any work happening in that space?

In theory you could use a protocol like this, one where the tools are specified in the page, to build a human readable but structured dashboard of functionality.

I'm not sure if this is really all that much better than, say, a swagger API. The js interface has the double edge of access to your cookies and such.

We're building an app that automatically generates machine/human readable JSON by parsing semantic HTML tags and then by using a reverse proxy we serve those instead of HTML to agents

As someone heavily involved in a11y testing and improvement, the status quo, for better or worse, is to do it the other way around. Most people use automated, LLM based tooling with Playwright to improve accessibility.

Chris Shank & Orion Reed doing some very nice work with accessibility trees. https://bsky.app/profile/chrisshank.com/post/3m3q23xpzkc2u

I tried to play along at home some, play with rust accesskit crate. But man I just could not get Orcas or other basic tools to run, could not get a starting point. Highly discouraging. I thought for sure my browser would expose accessibility trees I could just look at & tweak! But I don't even know if that's true or not yet! Very sad personal experience with this.