Remix.run Logo
▲ CharlieDigital a day ago

You can be productive and guide AI to operate in a more compliant way: by writing down basically everything you know, every heuristic you have, documenting the correct patterns, documenting the framework level approach, explicitly spelling out what's good code, writing down detailed specs for an agent to work on, writing the test plan, etc.

A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output.

A concrete example: I wrote a small library[0] (MIT, OSS) that improves how agents write unit and integration tests. This isn't product code, but this makes the agent better at writing the tests it uses to verify its work. (Tests become denser, more semantic, less context, type-checked setup, etc.). You can't expect an agent to do this by itself; the human needs to produce the scaffolding.

[0] https://github.com/zeeq-ai/zeeq-biome

▲Terr_ 21 hours ago | parent | next [-]

> You can be productive and guide AI to operate in a more compliant way: by writing down basically everything you know, every heuristic you have [...]

If its ends up being really that straightforward, it sounds less like engineering a product and uncomfortably close to building your own to corporate-owned replacement so that you can be fired.

▲boorang a day ago | parent | prev | next [-]

agree- this is how i name variables, this is how we handle third party APIs, this is how we handle caching and here are some heuristics for when we want to cache and when we want to avoid it, etc. etc. all coded up in a nice documentation tree with links in the agents.md that say "when integrating with third party APIs, read docs/integrating-with-third-party-apis.md".

i have written alot of this type of documentation as an architect for human teams, and i find it works for agents as well.

▲skydhash a day ago | parent | prev | next [-]

> A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output.

This basic reasoning is wrong and leads to “orchestration porn”. Thinking that you can laid down rules to automate software production like it’s a factory pumping out products.

What you forget is that software are grown iteratively, not guided down some assembly line. Every patch is a revision of what the software should do, not the next step of a workflow.

So stuff like patterns, architecture, tests,… are helpers for communicating the purpose of the software. They’re not really relevant to the software itself as the computer would be happy with just a single file.

You have a software that’s needed to fulfill some needs, you have a team of people where each brings expertise to build it. Workflows are built so that everyone how to contribute, what they can depends on, and what others need. When there’s a failure, we add the missing information. When there’s need to do some changes, we inform everyone impacted.

Everything is information and communication related. Information about the system and communication between everyone that interacts with the latter.

So devs do not think their job is to produce code. It is to take requirements written in ambiguous natural language, sometimes incomplete and contradictory, and to design a solution that is formally written while taking care that all statements are coherent.

Making things coherent (aka correct) is what takes the majority of our time. The other is gathering requirements. The actual typing of code is pretty much instinctive, just like you don’t check the orthography of words like “concrete” or “guide”, and rely on the spellchecker for more difficult one.

The “orchestration porn” part is that there’s no mention anywhere on how you think correctness and be improved or even maintained as part of your tool.

▲chickensong a day ago | parent | next [-]

Your argument seems to be about refuting a generalized one-size-fits-all approach, which can have its place, but there's nothing preventing per-project "orchestration porn" that works very well. Bind the harness to the project and your agent will help you with the non-code tasks as well.

> there’s no mention anywhere on how you think correctness and be improved or even maintained as part of your tool.

The first paragraph is literally a list of examples, followed up by a specific library that helped the OP. The paragraph you quoted explains where the maintenance lies.

▲Fr0styMatt88 a day ago | parent | prev [-]

I view the thing I’m ultimately providing to an AI agent is ‘executive function’, in the sense of how it’s used when talking about ADHD; one of the core executive functions is retrieving the right skills and/or information to use, at the point of performance.

AI agents don’t have this in a sense; or they do, but it’s average and stochastic. We have to be in the loop to provide judgement and manage the agent’s context; the content of that context steers the agent in the direction that our judgement tells us the development should go, to meet the various requirements. Some of those requirements are around the software’s function, some around maintainability, some around understandability, etc.

I think scaffolding is good at decreasing the amount of repeated mistakes and ensuring the agent can request and receive information efficiently. I think software factories are too self-referential; I’m not set on that opinion though and like I’ve said in another comment, I will fall down this rabbit hole at some point just to see what it’s like.

So many times I’ve wondered “you know how to do this, so why didn’t you just do it the first time?” — the old “Oh you’re absolutely right! That’s a much better way to do it!”.

It’s not that the model doesn’t have the knowledge, it’s that it doesn’t access it at the time it is needed. Part of my job now is to collaborate with the models and the tools.

▲skydhash 20 hours ago | parent [-]

From my experience there's always two parts in working with code. The deterministic part where you can derive information from the code itself and do some action, mostly used by static analysis tools (type checker, linting,...) and there's another much harder to define part like "this part of the code is unreadable and/or does not mean what the requirement says".

The first is solved by deterministic tool as code is already formal, so the grammar is very strict and fully defined. But the second part is entirely human dependent of what the end result should look like, even when the same semantics is being kept. That's why two people asked to implement the same task will code it differently even if from the point view of the machine, they're equivalent.

Maintainability and readability is not about software correctness. It's about how the human mind can absorb what the software should do, aka the specs embedded in the code, which is just formal notation. If it's easy, then the software correctness is easy to check and validate. And when the requirements changes, it's easy to apply the needed modifications.

So asking agents to improve upon that is like asking an agent to write an email. It's an alienation of human-human communication.

▲xmcqdpt2 a day ago | parent | prev | next [-]

Do you have eval testing? Benchmarks? Deterministic model checkpoints? Deterministic harness versioning?

Building "orchestration" without being able to deterministically measure improvements is not engineering, it's cargo cult.

▲CharlieDigital 2 hours ago | parent [-]

I don't know about you, but I watch my agents work and spot check work.

I also have a set of tools[0] specifically built to address this by surfacing agent usage of context and harness usage of tools.

Do I version? In some cases, yes. The tool I built for our team has configurable code review agents that trigger both in PRs and "in-the-loop" for the agent (exposes the same remote reviewer as MCP). It has an option to build a dataset to backtest changes to the code review prompts[1] and prompt history is saved as are dataset versions.

[0] https://zeeq.ai/

[1] https://zeeq.ai/docs/managing-zeeq/agent-code-reviews#backte...

▲haukebri a day ago | parent | prev [-]

Agreed on writing the test plan down. The harder part for me was what comes out of it. The tests an agent writes in my repo pass and protect nothing: copy text asserted as a fixed string, a component mocked away and then asserted on, three tests checking the same thing in different words. Coverage goes up and real breakage walks straight through.

Deleting first is what changed it for me. I go through the suite, cut everything that doesn't map to a flow a user actually goes through, then keep a few that drive the deployed app end to end. Fewer tests, noisier when one fails, and I trust them now.

The test plan is still the bit I can't hand over, because deciding what a flow is supposed to do is where I get it wrong myself.