| ▲ | CharlieDigital a day ago | ||||||||||||||||||||||
You can be productive and guide AI to operate in a more compliant way: by writing down basically everything you know, every heuristic you have, documenting the correct patterns, documenting the framework level approach, explicitly spelling out what's good code, writing down detailed specs for an agent to work on, writing the test plan, etc. A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output. A concrete example: I wrote a small library[0] (MIT, OSS) that improves how agents write unit and integration tests. This isn't product code, but this makes the agent better at writing the tests it uses to verify its work. (Tests become denser, more semantic, less context, type-checked setup, etc.). You can't expect an agent to do this by itself; the human needs to produce the scaffolding. | |||||||||||||||||||||||
| ▲ | Terr_ 21 hours ago | parent | next [-] | ||||||||||||||||||||||
> You can be productive and guide AI to operate in a more compliant way: by writing down basically everything you know, every heuristic you have [...] If its ends up being really that straightforward, it sounds less like engineering a product and uncomfortably close to building your own to corporate-owned replacement so that you can be fired. | |||||||||||||||||||||||
| ▲ | boorang a day ago | parent | prev | next [-] | ||||||||||||||||||||||
agree- this is how i name variables, this is how we handle third party APIs, this is how we handle caching and here are some heuristics for when we want to cache and when we want to avoid it, etc. etc. all coded up in a nice documentation tree with links in the agents.md that say "when integrating with third party APIs, read docs/integrating-with-third-party-apis.md". i have written alot of this type of documentation as an architect for human teams, and i find it works for agents as well. | |||||||||||||||||||||||
| ▲ | skydhash a day ago | parent | prev | next [-] | ||||||||||||||||||||||
> A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output. This basic reasoning is wrong and leads to “orchestration porn”. Thinking that you can laid down rules to automate software production like it’s a factory pumping out products. What you forget is that software are grown iteratively, not guided down some assembly line. Every patch is a revision of what the software should do, not the next step of a workflow. So stuff like patterns, architecture, tests,… are helpers for communicating the purpose of the software. They’re not really relevant to the software itself as the computer would be happy with just a single file. You have a software that’s needed to fulfill some needs, you have a team of people where each brings expertise to build it. Workflows are built so that everyone how to contribute, what they can depends on, and what others need. When there’s a failure, we add the missing information. When there’s need to do some changes, we inform everyone impacted. Everything is information and communication related. Information about the system and communication between everyone that interacts with the latter. So devs do not think their job is to produce code. It is to take requirements written in ambiguous natural language, sometimes incomplete and contradictory, and to design a solution that is formally written while taking care that all statements are coherent. Making things coherent (aka correct) is what takes the majority of our time. The other is gathering requirements. The actual typing of code is pretty much instinctive, just like you don’t check the orthography of words like “concrete” or “guide”, and rely on the spellchecker for more difficult one. The “orchestration porn” part is that there’s no mention anywhere on how you think correctness and be improved or even maintained as part of your tool. | |||||||||||||||||||||||
| |||||||||||||||||||||||
| ▲ | xmcqdpt2 a day ago | parent | prev | next [-] | ||||||||||||||||||||||
Do you have eval testing? Benchmarks? Deterministic model checkpoints? Deterministic harness versioning? Building "orchestration" without being able to deterministically measure improvements is not engineering, it's cargo cult. | |||||||||||||||||||||||
| |||||||||||||||||||||||
| ▲ | haukebri a day ago | parent | prev [-] | ||||||||||||||||||||||
Agreed on writing the test plan down. The harder part for me was what comes out of it. The tests an agent writes in my repo pass and protect nothing: copy text asserted as a fixed string, a component mocked away and then asserted on, three tests checking the same thing in different words. Coverage goes up and real breakage walks straight through. Deleting first is what changed it for me. I go through the suite, cut everything that doesn't map to a flow a user actually goes through, then keep a few that drive the deployed app end to end. Fewer tests, noisier when one fails, and I trust them now. The test plan is still the bit I can't hand over, because deciding what a flow is supposed to do is where I get it wrong myself. | |||||||||||||||||||||||