I'm just wondering (without really being an AI fanboy, but using it daily), isn't the whole challenge just more prompting, such as instructing the agents to analyze the codebase, propose refactoring approaches, etc? Maybe it's a bit more sophisticated prompting than a non-technical person would do but prompting nonetheless.

I'm asking because I have read these comments here and there about fixing the mess AI made. I know what mess it usually creates and it doesn't get better with the new models. But I also know that whatever is created by AI kind of works more often than not, and once something breaks, it's just one prompt away to fixing it.

▲

jamestimmins 2 days ago | parent [-]

In my experience, the problem is that once a codebase passes like ~50,000 lines, it's too big for AI to make good architectural choices (this is controversial, but I have yet to see evidence otherwise).

So what I do is mostly use the AI to understand things like architecture, encapsulation, deployment pipelines, types, migration structure, etc.

It's a lot of "Audit the callers of X and give me a report" as well as reading code. I'm looking for stuff that doesn't pass my smell test, especially relating type types/interfaces. I look at the AI recommendations, but they're only useful once I've scoped them down to more granular questions, such as assessing the interfaces between specific layers of a pipeline.

That process is about understanding the shape of the solution that exists and then trying to formulate what shape it should have.

Then it's a lot of identifying a very boring, vanilla architectural target and moving towards it in phases, using the strangler fig pattern pretty aggressively. I review every PR, but mostly at the level of architecture rather than every specific line.

▲

rhdunn 2 days ago | parent [-]

The codebase size will be limited to the LLM context window size. Once the codebase grows larger than the context window, some of the codebase will have to be removed from the context and thus not be known to that LLM instance.

There's also a well documented phenomena [1] where LLMs struggle to recall information in the middle of the context.

[1] https://charlesanthonybrowne.medium.com/how-llms-end-up-igno...

▲

jamestimmins 2 days ago | parent [-]

Good post.

Yeah over the past year I've become convinced that humans' limited working memory is actually a superpower, because it allows us to avoid this type of problem.

	▲	judahmeek a day ago \| parent [-]
		Isn't it less that we avoid the problem of limited working memory & more that we are constantly confronted by it and so we have had to devise processes for quickly organizing data by relevance to immediate & long-term goals, forgetting information that is no longer relevant, and creating new working memory chunks?