> For example I created a script that handles sending a message using Telegram reading the screen and tapping things using the accessibility service, it feels like magic.

Are you disclosing to people that they are interacting with an agent? If I learned a friend is using an agent to communicate with me they would get blocked right away. It’s really rude.

Also, you can do that via telegram API, no? no need for such an overcomplicated system

▲

yomismoaqui 13 hours ago | parent [-]

No, I just use it to write message via telegram dictating by voice.

Of course I can do it by API but the nice thing here is that with with a script + accessibility service I can automate any app on my phone, no need for them to provide an API.

The only gotcha is that you cannot use this when your phone is blocked, but seeing the phone do things "on its own" with a voice command seems like magic.

▲

dgellow 12 hours ago | parent | next [-]

But you said “reading the screen”, why would you need to read the screen if you’re using voice dictation?

What you’re describing doing (including the screen reading) is something that doesn’t require AI in any way, I’ve done similar things on my n900 early 2010s using pure bash. And I believe that can be done with standard automation like the Shortcut app on iOS.

It’s as magic as a bash script, but nothing revolutionary

	▲	yomismoaqui 12 hours ago \| parent [-]
		Reading the screen in this case is the agent seeing what is on the screen so it can interact with it using the acessibility service. I provide the agent with a hirearchical text representation of what's on the screen extracted from the accessibility tree provided by Android. It's crazy what an agent can do if you give it tools to see what is on the screen and other tools to tap, write, etc. The latest LLMs sure are well trained to interact with a phone. For example if I ask the agent to turn on the flashlight if swipes down to show the flashlight button and taps it, they sure know what they are doing.

▲

happyopossum 12 hours ago | parent | prev [-]

> No, I just use it to write message via telegram dictating by voice.

Umm, can't your phone do that natively without a convoluted cloud-based agent?

>The only gotcha is that you cannot use this when your phone is blocked, but seeing the phone do things "on its own" with a voice command seems like magic.

So you've got a more complicated way to do a basic task, with more restrictions - why can't you just tell your device to send a message using telegram with your voice using its native voice assistant?

	▲	yomismoaqui 12 hours ago \| parent [-]
		Try telling Siri or Google Assistant to open some specific app A, extract some information from it then open another app B and extract some other things from it, cross reference them, generate a result and send it to both your friend C using Telegram and your friend D using Signal. With this agent I can do it. Then seeing that it has done this well I can create an script that does this without calling the LLM in a repeatable deterministic way. This is the usecase I'm describing.