Ok, human produced wall of text incoming, hope you read it later ;)

The core of an agent is an agentic loop, that is, you provide some tools to the LLM and you have a for loop that sends the user request to the LLM and if it wants to invoke some tools you do it and send the results aback to the LLM (and loop again). Also if the LLM returns some text you show it to the user. Then when the LLM has returned the last answer and there are no more tools to call you just exit the loop. This is all there is, I learned about it from this article https://ampcode.com/notes/how-to-build-an-agent.

Right now I provide the agent some core tools that are always available from the start:

- spawn_agent: The agent can create a subagent to do some task using another context, this way you can for example launch new agent to do some long thing and get just the result without filling the root agent context with junk.

- load_skill: the agent can load a skill by its id. The lists of available skills is provided on the system prompt this way the agent can dynamically load something like "websearch" if it needs it to provide an answer to the user's request.

In short: skills are just markdown with a description on their frontmatter section, this same description is what is used on the system prompt to give the LLM info about that this skill does. Also the skill has some tools that are enabled when that script is loaded. For example some tools on the skill that uses accessibility service to control the phone:

- search_app_package: returns the package id from some text (e.g. search for "Whatsapp" get "com.whatsapp")

- launch_app: launch app using package name

- read_screen_content: returns a textual representation of the accessibility tree on screen (that is, you can read all things on screen and see which you can interact with)

- find_element: finds element by text with optional type and flag for exact match.

- click_element: clicks on element by the id returned by find_element (or an id from read_screen_content)

- etc...

Then as you can imagine using the LLM to call these kind of tools is slow, non-deterministic and it costs money, so the next step is allowing the agent to create scripts in JS that can invoke these tools. The typical use case is asking the agent to do something like sending a message using the whatsapp application and seeing it fumble around while it opens the app, taps things and writes text on the right input text (hopefully). Then in the same chat you ask the agent to create a script to automatize this and it does something like this (simplified JS pseudocode):

  result = await ("device.search_app_package", {text: "whatsapp"})
  await call("device.launch_app", {package: result.package_name})
  sleep(2000)
  result = await call("device.read_screen_content")
  ...

There are more details about the agent but this is the gist of it, ask about anything else if you want.