Thanks! Everything is overly complicated in this space. It's probably far easier than you think. The open secret is it's just a loop that POST [provider]/chat/completions.

Elixir is particularly well suited here. In Elixir this is a genserver doing http posts and reacting to the token stream. The LiveView chat gets messages from the genserver agent regardless of where it is on the planet, and the agent also communicates with the phoenix channel websocket talking to the IDE machines with regular messages, again anywhere they are on the planet.

I talk about this quite a bit in my ElixirConfEU talk and distill things down: https://youtu.be/ojL_VHc4gLk?si=MzQmz-vofWxWDrmo&t=1040

Wow, thank you for the link, it clarified how tool calling and "choice-making" works.

It's like helping LLMs use a computer; like building an interface for it.

Ok, this is enough to get me started.