> I took a photo of the clock, sent it to hermes,

How precisely do you do this? I'm interested in setting up a similar system, and I keep getting tripped up on how best to share information with the agent in a way that doesn't compromise security but is still easy to use.

Tailscale on a phone, on a server, and then whatever you and your agent come up with. I had a web app I built to talk to it. One day I wanted to share files and I just told it to implement that feature.

I don't use much off the shelf anymore. It's too easy to make my own hyper customized versions.