This analogy only works if you’re ready to commit _only prompts_ and have those prompts do a clean “compile” of your production runtime everytime you pull.

Are you ready to do that?

▲

Kim_Bruning 3 days ago | parent | next [-]

It turns out that this almost works. You'd probably want to commit explicit design documents btw, not every last prompt.

FWIW, when people try this out in practice, they tend to commit the design docs alongside the generated tests and the generated code.

Mutating the docs then leads to mutations in the code.

Several friends and I have workflows that look a bit like this, but I know for sure mine isn't 100% strict(yet?), and ymmv. Everyone is still figuring out the best practices I think.

▲

jtrn 3 days ago | parent [-]

Thank you for the first sensible response that actually engages with the point I made. And your answer is better than mine, and less angry.... :P

The question from glouwbug was just taken for granted to be "no, and therefore the analogy fails." And if I understand your answer, it's basically: "no, but we're visibly closer every quarter, and here's what the intermediate state looks like, and we might even get there"

The interesting thing in all of this to me is what must happen for the same spec to be deterministically certain to generate the same software. Could you delete the code, regenerate from docs alone, and trust the result? And obviously.... not yet. In practice, workflows drift, sometimes you patch the code directly because it's faster, and now code and docs have not been properly updated.

But the entire flow and concept of: [Spec] -> [AI/Harness] -> [Finished software], and how we increasing determinism in that flow, is just immensely interesting to me.

▲

glouwbug 3 days ago | parent | next [-]

Right, but hasn't generating the perfect spec always been the _exact_ problem of software development, regardless of language / LLM interface?

▲

3 days ago | parent | next [-]

[deleted]

▲

jtrn 3 days ago | parent | prev [-]

Yes, I agree with this fully.

But what’s interesting now is that the degree of determinism is increasing, as the community as a whole keeps refining the individual parts. The importance of spec becomes more obvious when the feedback loop speeds up, from spec to running software. That already made a huge difference in how many think about spec. There are multiple GitHub projects that are "spec only," where the goal is to spec it out in such a way that the software one wants is the inevitable result, if you just input the spec into a AI/harnes.

And the AI gets better, and the harnesses get better. So at some point we probably live in a reality where we can say: "If you spec out the software you want in this specific way, and add in these guidelines in AGENTS.md, and use X AI with Y harness, you almost certainly get identical software out the other end".

And yes "almost" is faaaaar from "always identical output". But the fact that we are even in the game of increasing determinism, in the Spec->AI->Software flow, is just mind-blowingly cool to me.

That is a detour from my main point, though, that on one specific level of analysis (can we move up one level of abstraction and lose some detail understanding, but gain more in productivity), AI, compiler, software frameworks, are all examples of the answer being: Yes.

And I do agree that we need to mitigate the damage that people with less experience can do because they don't know what pitfalls to avoid. But I would rather we focus on fixing that by improving the AI and harness, than the people that just keep saying that "AI is bad". In the same way I would rather make a tractor safer to use, not just complain that it's dangerous because someone drove it into the lake. Because the goal is not to make the perfect deterministic output from a compiler. that's just a step towards the real goal, which should, in my mind, be to help other people solve problems and do useful stuff. In the same way that the goal of the tractor is not to just plow the field, but to plow the field as fast and efficiently as possible so we can feed ourselves.

Wall of text because this topic has been bothering me for a while now, and Im using this thread to sort out my own thinking on it.

	▲	Kim_Bruning 3 days ago \| parent [-]
		> you almost certainly get identical software out the other end I think you'd get a set of convergent solutions, where the actual implementations might be very different. This also explains why you might want to keep more artifacts than just the design. There might be different performance overall, or interesting properties/side effects on axes that were not defined in the spec. Possibly -at the limit- a theoretically 'perfect' spec would be 100% deterministic (eg. by defining the final object code down to the bit) but that wouldn't be a useful operation I don't think? The other approach to getting bit-perfect output would be to aim for deterministic model runs. But this has numerous issues including making the model less "creative" or "smart" and anyway might not be the thing you actually want (to wit: a working artifact) .

▲

beej71 3 days ago | parent | prev [-]

I think your FORTRAN analogy depends on the assertion that a design doc is some kind of technical abstraction, like a compiler or a for-loop. And there's a lot of disagreement with that premise. :)

▲

jtrn 3 days ago | parent | prev [-]

See my response to the other comment if you don't understand that analogy is not the same as "identical".

And to state that "This analogy only works if ...." is just PATENTLY wrong. The analogy works fine if you say that it compares analogous situations. Like if we focus on some encumbrance complaining that "kids these days are too stupid because they don't understand the fundamentals like I do" or "These new tools that make it easier for stupid people, not smart people like me, to make stuff is dangerous because they don't know what they are doing". That's just a few of MANY analogous observations we can make for the two situations. But I guess you think that only the thing you care about is the only thing that exists.

And in the end, everybody who complains like this is just going to be shown to be just as mistaken as all the people who complained that "people who don't code in assembly are dangerous!" And it's just marvelous to watch it play out slowly over the last couple of years. And we are just a couple of years in. I'm just making a note of everybody who is mistaken, as a study in denial and biased thinking. The end for all of this was obvious after Opus 4.6 hit. And it's just getting more and more obvious with each model release and harness improvement. This is a gold mine for studying flawed thinking.