It turns out that this almost works. You'd probably want to commit explicit design documents btw, not every last prompt.

FWIW, when people try this out in practice, they tend to commit the design docs alongside the generated tests and the generated code.

Mutating the docs then leads to mutations in the code.

Several friends and I have workflows that look a bit like this, but I know for sure mine isn't 100% strict(yet?), and ymmv. Everyone is still figuring out the best practices I think.

▲

jtrn 3 days ago | parent [-]

Thank you for the first sensible response that actually engages with the point I made. And your answer is better than mine, and less angry.... :P

The question from glouwbug was just taken for granted to be "no, and therefore the analogy fails." And if I understand your answer, it's basically: "no, but we're visibly closer every quarter, and here's what the intermediate state looks like, and we might even get there"

The interesting thing in all of this to me is what must happen for the same spec to be deterministically certain to generate the same software. Could you delete the code, regenerate from docs alone, and trust the result? And obviously.... not yet. In practice, workflows drift, sometimes you patch the code directly because it's faster, and now code and docs have not been properly updated.

But the entire flow and concept of: [Spec] -> [AI/Harness] -> [Finished software], and how we increasing determinism in that flow, is just immensely interesting to me.

▲

glouwbug 3 days ago | parent | next [-]

Right, but hasn't generating the perfect spec always been the _exact_ problem of software development, regardless of language / LLM interface?

▲

3 days ago | parent | next [-]

[deleted]

▲

jtrn 3 days ago | parent | prev [-]

Yes, I agree with this fully.

But what’s interesting now is that the degree of determinism is increasing, as the community as a whole keeps refining the individual parts. The importance of spec becomes more obvious when the feedback loop speeds up, from spec to running software. That already made a huge difference in how many think about spec. There are multiple GitHub projects that are "spec only," where the goal is to spec it out in such a way that the software one wants is the inevitable result, if you just input the spec into a AI/harnes.

And the AI gets better, and the harnesses get better. So at some point we probably live in a reality where we can say: "If you spec out the software you want in this specific way, and add in these guidelines in AGENTS.md, and use X AI with Y harness, you almost certainly get identical software out the other end".

And yes "almost" is faaaaar from "always identical output". But the fact that we are even in the game of increasing determinism, in the Spec->AI->Software flow, is just mind-blowingly cool to me.

That is a detour from my main point, though, that on one specific level of analysis (can we move up one level of abstraction and lose some detail understanding, but gain more in productivity), AI, compiler, software frameworks, are all examples of the answer being: Yes.

And I do agree that we need to mitigate the damage that people with less experience can do because they don't know what pitfalls to avoid. But I would rather we focus on fixing that by improving the AI and harness, than the people that just keep saying that "AI is bad". In the same way I would rather make a tractor safer to use, not just complain that it's dangerous because someone drove it into the lake. Because the goal is not to make the perfect deterministic output from a compiler. that's just a step towards the real goal, which should, in my mind, be to help other people solve problems and do useful stuff. In the same way that the goal of the tractor is not to just plow the field, but to plow the field as fast and efficiently as possible so we can feed ourselves.

Wall of text because this topic has been bothering me for a while now, and Im using this thread to sort out my own thinking on it.

	▲	Kim_Bruning 3 days ago \| parent [-]
		> you almost certainly get identical software out the other end I think you'd get a set of convergent solutions, where the actual implementations might be very different. This also explains why you might want to keep more artifacts than just the design. There might be different performance overall, or interesting properties/side effects on axes that were not defined in the spec. Possibly -at the limit- a theoretically 'perfect' spec would be 100% deterministic (eg. by defining the final object code down to the bit) but that wouldn't be a useful operation I don't think? The other approach to getting bit-perfect output would be to aim for deterministic model runs. But this has numerous issues including making the model less "creative" or "smart" and anyway might not be the thing you actually want (to wit: a working artifact) .

▲

beej71 3 days ago | parent | prev [-]

I think your FORTRAN analogy depends on the assertion that a design doc is some kind of technical abstraction, like a compiler or a for-loop. And there's a lot of disagreement with that premise. :)