Running tests that the agent created for its own hallucinations? Optimize using another hallucinating agent?

Which tests and optimizations do you propose to run after a night of supervised work when one of main things that all agents keep doing is "load all records from db , and filter them in memory"? It's now become so bad, I had to literally vibecode a separate linter for this. And that's just one of the problems.

we don't have master AI that can create a great product out of a mediocre prompt.

but we do have sufficient AI to make a great product out of a great prompt.

garbage in -> garbage out hasn't gone anywhere.

so: much like to anyone that blindly complains that their compiler hates them : if you actually want help, provide information. If you just want to complain that the compiler is mean, scream at the sky.

plenty of people have figured out how to get this to work; more than enough to confirm that a straight <gambling-machine>/<hallucinatory-psychopath>/<random-number-generator> analogy is too simplistic to explain what we're working with.

>It's now become so bad, I had to literally vibecode a separate linter for this.

You see, there's your problem right there. You're vibe coding, which by definition literally means you're unwilling to look at the generated code. That's not what successful ai assisted software developers are doing. YOU HAVE TO READ THE CODE. Refusing to do that means you're not a serious programmer, you're outsourcing your thought and design and implementation, trying to get something for nothing by taking the easy way out, and you're going to get terrible results no matter what prompts you "engineer". There ain't no such thing as a free lunch (yet).

And while we're at it, to elaborate on what serf said: people mindlessly parroting terms like "stochastic parrot" to criticize llms without having read the actual paper that coined the term and understanding what it really claimed and how other papers responded to it means you're just a human stochastic parrot no better than what you're criticizing -- at least the llm has read all those papers and understands what "stochastic parrot" actually means in context. Ask it, it will be glad to explain!

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (FAccT 2021)

https://dl.acm.org/doi/10.1145/3442188.3445922

If this is your problem, I hate to say it, but it’s a skill issue on your side. I didnt even start developing software until LLMs even though I know all the primitives quite well and have a strong grasp of architecture. ADHD, perfection, and focus got in the way. I’m writing a pretty complex application and I don’t have the problems you’re running into. I regularly have my code reviewed by professional devs to ensure I’m not just vibecoding into slop. I’m not. I also read my code regularly and do a lot of exercises and courses to keep learning as well.

So I dunno what to say, except it’s possible to write really solid code with LLMs.