I think we’re still figuring out the right abstraction for offering agents as a product.

- LLMs are a great foundation but building your own harness is a huge undertaking, a deep rabbit hole.

- There are harnesses available as open source libraries but that’s still coupled to an environment. Where does the state persist? Like maybe I’m a Cloudflare worker and don’t even have a file system.

Agent as a service like this lets you plug in the tools it needs to be whatever kind of agent you want. But they still get to encapsulate and continue to iterate on the really deep parts of the harness that all agents need like memory and context management.

That said, my money right now is not on the offerings from OpenAI and Anthropic because they’re stuck using their own proprietary frontier models and those aren’t actually the best choice for most agents right now. A competitor who is not an LLM lab gets their pick of the market at any given moment. Like you’d want to be using GLM 5.3 Flash right now for most things agentic.

> LLMs are a great foundation but building your own harness is a huge undertaking, a deep rabbit hole

I’ve been doing this for the past few months. I started with a server where I ran pi in tmux and then used that to build an LLM gateway and agent session manager, then built deterministic workflows using bash scripts and a skill/script distribution system. The app works on desktop, mobile and web and it works great. Non technical colleagues are using it to build and ship real software and it’s cheap AF even using API pricing because it works well with Luna or deepseek.

> building your own harness is a huge undertaking, a deep rabbit hole.

I eventually gave up on this task. It's not possible to fight OpenAI or Anthropic's engineering teams. Their reasoning models have all kinds of undocumented back door access to the base models that you'd never be able to replicate from the outside. Even if you had full access you would not have the engineering man hours or experience to keep up.

I think this Agents API thing is a step too far, but Chat Completion is too cold now. Something approximating Responses API seems like the happy medium. You still get most of the control with the only blackbox part being the reasoning loop / tokens. Building agents using the GPT5.6 family w/ Responses API feels pretty close to Star Trek computer shit to me. I thought I was being clever with my DIY contraption on top of chat completion, but it wasn't even close. I have embraced the reality that I will need to use opaque reasoning tokens to give my clients the experiences they are paying me to provide.

I think the abstraction is only part of the problem. The other part is that all these companies offering ai products are deeply untrustworthy, and I don’t want to let them any further into my stack than I have to. Claude code and codex are great because they are lightweight, and operate on top of the rest of my tools with little to no change needed, so they can be eliminated or migrated away from with zero cost. They’re not a dependency of anything. And that’s as much as I’m willing to trust OpenAI or Claude.

The best answer I’ve come to thus far is the model we (estuary.dev) are building out now: offering mcp.estuary.dev with tools for creating a sandbox with our CLI pre-installed, a tool for requesting that a tightly scoped access token be injected into a named sandbox file (this is the approval gate), and a tool for executing arbitrary commands in the sandbox (presumably our flowctl CLI, but let the model rip).

The intent is that anybody can drive it from Claude/ChatGPT/Pi on their phone after MCP sign-in (oauth), the model has full computer use capability, but we can also leverage it to build guided agent workflows in our own dashboard.

I built several harnesses in different products over the last two years. Fully agree with you that doing it right is a rabbit hole. Certain system properties that you almost always want in a harness used within a SaaS (for example) are non-obvious at the start and require certain architectural choices. It's easy to start down a path and then find a gap a couple days before launch.

Async tool calls, having the agent wait indefinitely for a human response, and showing a form or questions to the user via a tool call are a few common capabilities that come up that a product manager might miss at first.

This is why I've been building Nvoken. LLM agnostic, ergonomic SDKs, flexible tool call patterns, tenant and user-aware budget enforcement, etc.

I'd really appreciate any and all feedback on this! It gives you some free tokens on signup and it's super quick to try.

https://nvoken.com

There might not be a good abstraction. I've built a few harnesses for different types of workflows, and the details are so different I struggle to see a good abstraction. It's also not clear there should be - if you look at most complex software systems, it's a collection of smaller abstractions/tools/systems pulled together to achieve X.

Agree, as long as models are interchangeable, it doesn't make sense to be locked into a single lab's managed agent platform. You probably want to swap between models and own the agent state.

https://github.com/omnara-ai/omnara - this is a self hostable agent API that I'm working on. It stores the state of all agents in a postgres db you can easily query, rather than a local json file or sqlite file per agent.

I just have a slack bot running on a VM that sees a message and invokes pi.

It would be trivial for every request to clone a full lxd container and have all the tools and repos required if I wanted to allow it to do even more.

Not sure why anyone prefers to choose locked in options

Agents are the wrong paradigm entirely and have limited places where they actually belong.

Going off and searching the web isn't really it.

You need to create 'new worlds' where they can operate best - and even then constrain what it does.

> Where does the state persist?

Spider men meme of developers pointing at each other thinking "Not it".

I think things like onecli are the direction we will take. The secrets and state will be proxied api calls.

[dead]