But you said “reading the screen”, why would you need to read the screen if you’re using voice dictation?

What you’re describing doing (including the screen reading) is something that doesn’t require AI in any way, I’ve done similar things on my n900 early 2010s using pure bash. And I believe that can be done with standard automation like the Shortcut app on iOS.

It’s as magic as a bash script, but nothing revolutionary

Reading the screen in this case is the agent seeing what is on the screen so it can interact with it using the acessibility service. I provide the agent with a hirearchical text representation of what's on the screen extracted from the accessibility tree provided by Android.

It's crazy what an agent can do if you give it tools to see what is on the screen and other tools to tap, write, etc. The latest LLMs sure are well trained to interact with a phone.

For example if I ask the agent to turn on the flashlight if swipes down to show the flashlight button and taps it, they sure know what they are doing.