I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.

I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.

Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area

▲

loveparade 3 days ago | parent | next [-]

If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.

	▲	yehoshuapw 3 days ago \| parent \| next [-]
		that is true, but if the metric is what we want optimized, then that's fine. However it is more likely to be something which can be detached..
	▲	datsci_est_2015 3 days ago \| parent \| prev [-]
		Code quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.

▲

peder 3 days ago | parent | prev [-]

Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.

Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.

	▲	alecbz 3 days ago \| parent \| next [-]
		> that's primarily a human concern that's unrelated to the final output. It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
	▲	FuckButtons 3 days ago \| parent \| prev \| next [-]
		I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
	▲	stiiv 3 days ago \| parent \| prev [-]
		I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree. On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.