I had a similar experience, but it worked just well enough that I wanted to try my own similar experiment, but using a functional API in Elixir and giving the agent some introspection tools. I'm very much a novice to this space and learning as I go, but I've been able to make some interesting projects with the library so far, which is all I really want to do: https://smith.hexdocs.pm

If you want to build by hand, there is a Kino renderer to build your projects interactively in Livebook, which is actually surprisingly helpful for learning.