At scale, specs can only be vital to the degree to which their conformance testing is automated. Good specs should use a formal, runnable verification language. Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.

▲

seanmcdirmid a day ago | parent | next [-]

But conformance testing is where agents really excel at if you set things up right:

* Black-box testing: the agent writing the tests cannot see implementation and the agent writing implementation cannot see tests, they only agree on a spec and an interface (the minimum needed to write tests).

* Evaluate test coverage using code coverage, but when gaps are found communicate those gaps in terms of the specification.

Good specs should be grounded (complete and not ambiguous), they don't need to be formal. You should be able to re-run your agents when the spec changes on diffs to the spec, and if a change happens out of bad, you should have agents that go in and propose fixes to the spec. Since agents are doing deterministic codegen like a compiler would, this is all pretty straightforward.

You also need to consider public and internal specifications (the public specification being for reuse of the component), and you might test your integrating component with a test-double (built from the public specification alone) rather than the real component itself.

▲

sdevonoes a day ago | parent [-]

> Good specs should be grounded (complete and not ambiguous), they don't need to be formal

IMHO, this is a mistake. I guess we play with it because there’s isn’t anything better nowadays. Writing and maintaining “specs” in plain english is painful.

	▲	seanmcdirmid 21 hours ago \| parent \| next [-]
		Natural language is incredibly expressive and fairly easy to read and write. Pick your favorite formal specification language...and you can just express some properties in them, and they are mostly niche at that. If you increase the expressiveness of your formal language too much, it just becomes code, then you are back to square one. LLMs are also incredibly proficient in processing natural language; i.e. writing, maintaining, and using a spec written in plain english is actually fairly viable with a modern LLM.
	▲	crazygringo 20 hours ago \| parent \| prev [-]
		> Writing and maintaining “specs” in plain english is painful. The subject of this entire post is development with agents. Writing specs in English is how you do that. If you don't like it, then this is probably not the right article for you to be commenting on.

▲

crazygringo 20 hours ago | parent | prev [-]

Not true.

This is about ensuring that when AI's make future changes, they aren't just looking at the existing code and making assumptions about intent. They should always be pulling the specs to ensure that changes maintain compatibility with the specified intents.

> Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.

Not if the changes you're making are always to the specs, as opposed to the code. The whole point here is that you don't change the code, you change the specs, then approve the code that the LLM changes as a result. This way the spec should never diverge from the code.

AI absolutely changes the dynamic so that code doesn't converge from the spec. That's the whole point, and the whole point of committing the specs like code.

Automating testing is great too, of course, but that's not the full picture. It ensures formal compliance, but doesn't encapsulate anything about the spirit or purpose of why the design in a certain way. Good specs do. The purpose/motivation sections and engineering guidelines are some of the most important for an LLM. Which is what helps the LLM figure out how to then best modify the existing code when features need to be changed or added.

▲

skydhash 19 hours ago | parent [-]

> The whole point here is that you don't change the code, you change the specs,

> but doesn't encapsulate anything about the spirit or purpose of why the design in a certain way. Good specs do.

Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones. So they're always simplistic, because they ignore the cascading effect of implementation decisions.

Generating code with AI is rolling the dice every time said generation is done. Proper implementation happens because with making decisions and going down a path, backtracking if necessary when it's no longer working. Going with AI is breaking down that continuity because they restart from scratch everytime.

▲

crazygringo 19 hours ago | parent [-]

> Specs are meta solutions. They describe the general shape of the solutions by refusing to make any technical decision that would leads to incidental problems and thus only needs to focus on the essential ones.

Yes, this is a feature not a bug. Then, code review ensures that the actual code makes the requisite technical decisions in a desirable way. This is much faster.

> Generating code with AI is rolling the dice every time said generation is done. Proper implementation happens because with making decisions and going down a path, backtracking if necessary when it's no longer working. Going with AI is breaking down that continuity because they restart from scratch everytime.

False. You only do the initial code generation an initial time, and then update the spec to specify (at a high level, ideally) the constraints that are missing. But then you have the AI modify the existing code to meet the new spec, not start from scratch. AI does not "restart from scratch everytime", it takes an updated spec plus existing code and updates the code to match the updated spec.

It's still "making decisions and going down a path," which you call proper implementation. That's the whole point.

	▲	skydhash 18 hours ago \| parent [-]
		The reason people don't bother with specs is that they're extraordinarily verbose, ambiguous, and simplistic at the same time. Which is why we have invented formal notations, because they're short, have one definition, and enough to create the whole thing, if not to understand the why. The latter can still be done in prose. What a spec is truly useful for is that it lets you manipulate the solution at the idea stage, without tainting it with real world concerns. But to do it requires a lot knowledge about those real world concerns, so that your assumptions are not straight lies. This is good for iteration. But any implementation ossifies your specs, and while code are more malleable than most, there's a certain points when the specs should be discarded and the actual implementation being the source of truth for any further development. In any project, that is done with Change Request, which only outline the specific area of changes, assuming the rest as being immutable. > AI does not "restart from scratch everytime", it takes an updated spec plus existing code and updates the code to match the updated spec. Maybe you can share an example of that workflow and an actual argument that would make it sustainable. I've written the formal version of a spec (a suite of tests) and if there's one thing that would make the above idea ridiculous is how much wiggle room there is to follow the specs and be at the same time unusable.