Remix.run Logo
i_have_an_idea 6 hours ago

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.

hannofcart 5 hours ago | parent | next [-]

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.

I wrote down a more detailed post of the various linters I use here:

https://www.balajeerc.info/Use-Deterministic-Guardrails-for-...

saghm 2 hours ago | parent | next [-]

Sure, giving agents a deterministic way to run linters is good, but that already implicitly accepts the premise that code hygiene does actually matter. If you agree with that, then you probably also recognize that there are some things that you can't lint for because they're a lot harder to define, and why would you expect an agent to be able to figure that out today when they still need our intervention to even properly run linters?

rafaelmn 5 hours ago | parent | prev | next [-]

> Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of them - produces half legacy garbage.

Linters only handle trivial cases most of us already solved.

yoyohello13 an hour ago | parent [-]

Yeah, we have a big struggle with this. We have lots of legacy code that doesn't follow our latest design patterns intermixed with new code. The LLM picks up bad habits depending on what it pulls in to context first. We have AGENTS.md configured with the right way, but old style still slips in. We obviously need to update the old code but on the other hand if it ain't broke why touch it.

kstenerud an hour ago | parent [-]

I have the agent inject comments that mention that this particular code is legacy and must not be used as a reference, should not be cleaned up, etc. If you have a document that lists all of the reasons not to use or touch some code, the comments can simply be references to it.

    // LEGACY CODE, per docs/legacy_rules.md §14, §19
maccard 35 minutes ago | parent | prev [-]

> You can get the LLM to run a script which checks for all of these

Most of the time. Except for when it forgets to do it.

I think it’s funny that the solution is to use something that is not LLM driven to enforce it.

Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)

Yokolos 24 minutes ago | parent [-]

This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.

yoyohello13 5 hours ago | parent | prev | next [-]

I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.

jaggederest 4 hours ago | parent [-]

It actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, but the context for agents is taking up the same space and "attention" every time they're run, and they're creations entirely of context, so the quality and examples matter massively.

maccard 34 minutes ago | parent [-]

> actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing,

You’ve never had an agent completely lose the plot and forget/confuse its instructions due to the context filling up?

BobbyTables2 5 hours ago | parent | prev | next [-]

Feel the same way myself when working in messy codebases… At some point, the horrible patterns start to rub off…

NitpickLawyer 2 hours ago | parent [-]

Every time this subject comes up, there are a bunch of takes along the lines of "would you work on a codebase maintained by agents? they'll mess up the code". And I'm asking myself where these people work, because in 20+ years I've yet to see that pristine state of a project that keeps being pristine after the honeymoon greenfield phase, and 50+ people start working on it. Every project devolves in time, old stuff gets patched in a hurry, someone tries to make it better, learns why certain things were done a certain way, hits some undocumented client needs handled by some arcane combination of code + external systems, and so on. If anything, keeping track of what does what in a project is a task where agents can shine, if only in "ask" mode so you can figure out things quicker. Not to mention onboarding and stuff for new team members.

ramraj07 6 hours ago | parent | prev | next [-]

I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem.

The most useful discussion would be if we all read the paper and critique its methodology or results.

vetronauta 2 hours ago | parent [-]

I was reading your comment, disagreeing with it but still feeling why this is a good comment. It just occurred to me that this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon.

dnautics 6 hours ago | parent | prev [-]

i mean this is feeling too but im too paranoid and frequently do refactoring and code organization passes and never don't do it, so i cant say i know for sure there is a delta.

though people who complain that llms aren't that great strike me as the type to have messy code bases