This is going to sound incredibly silly but I have every HTMX component expose an AI summary with actions possible and then I have a small copy to AI button on the page which creates a new short-lived access token and copies into a prompt the token, and all the components as text.

When I give this even to relatively small-model agents, they use this initial seed to browse the site and do things very well.

The initial prompt suffices. After that most agents just use the HTML to navigate very well. I suppose I could have that description in an aria label if I wanted but it’s the same.

▲

pbronez 2 days ago | parent [-]

I'd be interested to read a blog post with more details about how this works.

	▲	arjie 9 hours ago \| parent [-]
		Sorry I missed your comment, Peter. I don't know that it's worth very much to describe. It's not sophisticated. Each component just has an attribute named data-ai-context and content that describes it. This needs no LLM. e.g. charts go into canvas, so their data-ai-context has a textual summary of the content (same as is fed to the charting library); buttons describe what they'd do. Then when you click the "Copy to AI", a piece of JS hits a backend to sign a 15 min token for the same user (as if they'd logged in but with a short token) and pulls all the data-ai-context attributes and gives you a prompt in your clipboard describing the site organization. It just so happens that doing that tiny first bit allows even stupid models to navigate the site. After that, advanced models also sometimes query for the data-ai-context and they can mimic the Copy To AI to not have to nav the site. e.g. content "Job Failure Rate: failed sync jobs as a share of jobs run, per connector per day, over 7 days. Day X: facebook=8 failed / 862 jobs; ga4=0 failed / 196 jobs;" It's really just an alt text element and bizarrely worked the first time I tried it with old Opus 4.5 and I've kept it. Perhaps the biggest use is giving the agent the short-term token.