Remix.run Logo
AdieuToLogic 2 days ago

> I've been using AI to contribute to LLVM, which has a liberal policy.

This is a different decision made by the LLVM project than the one made by Gentoo, which is neither right nor wrong IMHO.

> The code is of terrible quality and I am at 100+ comments on my latest PR.

This may be part of the justification of the published Gentoo policy. I am not a maintainer of same so cannot say for certain. I can say it is implied within their policy:

  At this point,  they pose both the risk of lowering the 
  quality of Gentoo  projects, and of requiring an unfair 
  human effort from  developers and users to review 
  contributions ...
> LLMs increase review burden a ton ...

Hence the Gentoo policy.

> ... but I would say it can be a fair tradeoff, because I'm learning quicker and can contribute at a level I otherwise couldn't.

I get it. I really do.

I would also ask - of the requested changes reviewers have made, what percentage are due to LLM generated changes? If more than zero, does this corroborate the Gentoo policy position of:

  Popular LLMs are really great at generating plausibly 
  looking, but meaningless content.
If "erroneous" or "invalid" where the adjective used instead of "meaningless"?
jjmarr a day ago | parent [-]

   I would also ask - of the requested changes reviewers have made, what percentage are due to LLM generated changes? If more than zero, does this corroborate the Gentoo policy position of "Popular LLMs are really great at generating plausibly looking, but meaningless content." 
I can only speak for my own PR, but most requested changes were related to formatting and other stylistic issues that I didn't fully grasp as a new LLVM contributor. e.g. Not wrapping at 80 characters, forgetting to declare stuff as const, or formatting the documentation incorrectly.

Previous codebases I've worked on during internships linted the first two in CI. And the documentation being formatted incorrectly is because I hand-wrote it without AI.

Out of the AI-related issues that I didn't catch, the biggest flaws were redundant comments and the use of string manipulation/parsing instead of AST manipulation. Useless comments are very common and I've gotten better at pruning them. The AI's insistence on hand-rolling stuff with strings was surprising and apparently LLVM-specific.

However, there was plenty of erroneous and invalid behaviour in the original AI-generated code, such as flagging `uint32_t` because the underlying type was an `unsigned int` (which wouldn't make sense as we want to replace `unsigned int` with `uint32_t`).

I prevented most of this from reaching the PR by writing good unit tests and having a clear vision of what the final result should look like. I believe this should be a basic requirement for trying to contribute AI-generated code to an open-source project but other people might not share the same belief.