"agent pipelines that [...] clean a messy [repository]"

This feels like a terrible approach, sufficient to condemn the entire study.

Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

▲

geraltofrivia 2 minutes ago | parent | next [-]

First author here. Please let me offer a clarification. Our notion of "clean" isn't to just ask the agent to write better code. Rather, we give it a list of 50-100s static analyzer rule violations (and code LOC), and ask to remove them. We then check if the rule violations are resolved.

Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1] (in production since 1+ year), and a recent agentic approach [2] to do the same work rather well to do this.

[1] https://www.sonarsource.com/solutions/ai/ai-codefix/

[2] https://www.sonarsource.com/products/sonarqube/remediation-a...

▲

ramraj07 6 hours ago | parent | prev [-]

Would you trust clean repos that are messed up by AI?

	▲	wgd 5 hours ago \| parent \| next [-]
		Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs. I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.
	▲	smallerize 5 hours ago \| parent \| prev [-]
		No, because the outputs will still be "in distribution" so to speak.