Remix.run Logo
KronisLV 4 hours ago

> Then cry as the ai reveals that it didn't actually do anything close to what it said it did.

If using AI to generate code, you told it generate some code, so it did. No amount of "You are an expert developer" or "Make no mistakes" will change the fact that it just generates tokens and has a limited thinking budget.

Adversarial review loops of N parallel agents looking at whatever characteristics you care about will make it better, even if it will Nx the tokens you need to achieve something, though in general it will be cheaper than N human reviewers (which you might not have).

Obviously you shouldn't forget about traditional tooling for formatting and linting, as well as static code analysis and having test coverage that approaches 100%. It might be annoying to do manually, but AI has no issues with refactoring code to make it more testable and eventually will catch some issues that way. It's never going to be perfect in the 1st attempt.

> Review the codebase is it production ready? I'm selling it for $1million dollars can it meet that standard.

This is far too vague though and will never be good, even sans AI. When it comes to AI, it will nitpick the fuck out of the codebase if you ask it to do and sometimes jump around between different approaches because neither is actually a good fit for the problem space (there might not be a good fit at all, just various tradeoffs). If you still ask it to find issues and there's nothing obvious, it will just make shit up in pursuit of being useful (RLHF).

When it comes to people, you will get various standards, from "It looks like Java, ship it" to "You should rework a quarter of your codebase because I read about this one approach in an authoritatively written book that you should also follow because I view it as dogma and will hold back your merge until it all works exactly like I want it to." (you get all sorts of people and personalities).

In my experience other people are no panacea either, nor is writing code all by myself. Fuck it, I'll take anything and everything to help me ship stuff that's good enough and on time (even if some/most? deadlines within the industry are made up). I'd argue that producing something that would pass most critique and could be considered "good code" (not "good enough") or even more broadly a "good product" might take about an order of magnitude more effort than most people and organizations actually can, or can budget for.

mosura 3 hours ago | parent [-]

> Adversarial review loops of N parallel agents looking at whatever characteristics you care about will make it better

Especially if different model providers. It becomes more like having team members that see things slightly differently.

ModernMech 39 minutes ago | parent [-]

It's like a variant of the halting problem though: given N agents in a review loop reviewing a codebase, will it ever terminate and say the thing is done and bug free?

It seems to me just from @codex review, given a codebase of any appreciable size, if you ask the agent to find things to fix, it'll find things to fix. And despite N agents agreeing that some code is ready to ship, I've still sat down to try it and nothing actually works as advertised.

So the need for better verification tools that are not AI is very urgent. Because the AI can be told to write a thing without mistakes, it does write code that compiles, N-agent AI review process fixes bugs and eventually the code is approved, and then run against an extensive suite of tests to prove certain functionality which all pass... and yet it still can be the case that nothing actually works in production.

KronisLV 23 minutes ago | parent [-]

> will it ever terminate and say the thing is done and bug free?

Will human reviewers, if you ask them to find more bugs? Will it be because there really are no bugs, or because they got tired and just can't see any?

How would you even figure out when the thing is done and is bug free? There have been bugs that have been in codebases for years and have only been discovered recently.

Would you trust static analysis tools that give you an all clear, when they themselves cannot encode the above desired state fully?

Seems like you'd need formal proofs or something, but at that point also for the frameworks and libraries, database, networking stack and the whole damn OS.

I'm all for good tools, just saying that we'll need a whole bunch of those.