Remix.run Logo
kimjune01 4 hours ago

I agree with iteration, but that's an orthogonal argument to how effective it is at turning prose to code. You are asking, LLMs are capable of translating a 835-page spec, but where does the spec come from? Follow the stream up from the built software product:

Binary < IR < Code < Spec < Prompt (aka goals)

For LLMs to faithfully fulfill the role of translating spec/prompt to code, it needs enough information about the problem/motivation and the constraints around it. Often, that's done by research or by importing proven modules. The human role left in this compilation chain is a precise-enough expression of the motivation, upstream of the built artifacts.

Edit For example, SWEbench Pro was supposed to be a measure for how good LLMs are at compiling prose into code. At least 15% of those tasks were underdetermined by ambiguous prose in the spec. Of the issues that are specific enough, Claude and Codex indeed do compile to functional code with self-written tests. Source: https://github.com/kimjune01/swebench-pro-audit