We built something similar[1], including integrated memory for debugging. It is very useful to have repeatable artifacts left behind every time you use an agent to accomplish a task.

The main design decision we took was to integrate with your existing agent instead of building a new one. Your harness, swamp, and you're off.

As an aside, building software for agents is incredibly fun.

1: https://swamp.club