I do not have the burnout but I certainly operate similarly to the author. I continue to be unable to establish a workflow where allowing the LLM to generate code that I review is faster than writing the code myself. Literally the only two ways out of this dilemma is to blindly trust what was generated or to generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. I just write the business logic myself and have the LLM do a lot of the rest. Boilerplate falls into the latter as well.

▲

jaggederest 2 hours ago | parent | next [-]

> generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario.

This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is the best way to keep AI on track and make those diffs you have to read clean and easy.

An even better way is to use a more strongly typed language and really lock it down, but you can use testing in any language. I feel like my background in TDD and "TATFT" has been secret sauce when working with AI

▲

rafterydj 2 hours ago | parent | next [-]

I see this get mentioned a lot but I still am skeptical that AI can generate tests we can trust more than any other code we know we cannot trust.

Yes tests are conceptually isolated and that helps, but I've personally seen unit tests get generated that are semantically incorrect - that is, they test the structure of the code (e.g. they can check function output types and values), but they can't know _why_ the unit tests need to be there, so the really really helpful tests never get generated. Not to mention the obvious issues with generated tests only testing is x = x, or needless redundant tests for the same thing, or them essentially testing basic features of the language.

▲

jaggederest 2 hours ago | parent | next [-]

You have to iterate on the tests, review and validate them, just like any other code, and if you generate a whole project's tests all at once the quality is abysmal, of course. I've been using a lot of old school data-driven testing techniques, where the harness is just code I review, and the data itself is e.g. json files and drives the system.

I actually have a public (AGPL) example here: https://github.com/pgdogdev/pgdog/tree/main/integration/sql - pgdog is particularly testable since it is trying for complete transparency, so you have a perfect oracle in hand via base postgresql, but it demonstrates the concept at least.

▲

fzeroracer 16 minutes ago | parent [-]

Then this falls into the exact same pit the OP mentioned, either you need to blindly trust that the LLM is generating tests that actually work, or you need extensive test coverage for your tests to ensure that your tests are actually testing.

	▲	jaggederest 14 minutes ago \| parent [-]
		It turns out that you don't actually need tests for your tests, because the code provides a baseline truth for the tests. You do, at some point, have to be epistemically sound enough to actually look for correctness in either the code, behavior, or tests. We unfortunately haven't fully unlocked completely solipsistic value generation yet. This is also part of why I like end to end tests that use actual UI flow, so I can watch it go by in slow mode before letting it loose fully automated.

▲

nzeid 2 hours ago | parent | prev | next [-]

Which is why test generation has to be carefully guided as well, and this is something at which I've incidentally been fast. Ultimately it's a constant battle between LLM handholding and doing things yourself.

▲

skydhash 2 hours ago | parent | prev [-]

I don't even care about tests being correct as you can still verify them even when tedious. What I care is that, more often than not, the shape of the solution is not fixed. Having unit tests for those can be extremely costly as when the changes happens, you have to change all the tests.

I've been burned by this in my honeymoon period with unit testing (pretty much the reason it ended). These days, I prefer broader scope of testing, especially user-facing part. The users may be other developers or end users. I only do unit testing for tricky algorithms or math formulae.

▲

jaggederest an hour ago | parent [-]

I want all the layers of the pyramid, eventually, but the top layers matter the most. I can't count the number of times my paranoid "make sure that customers can successfully pay us" end to end test suite has prevented the money faucet from being shut off. I install one perennially at any company I work at and they always pay for themselves surprisingly quickly.

	▲	skydhash 26 minutes ago \| parent [-]
		I’ve been involved in B2B (so no payment flow). But it’s basically the same with an handful of integration tests for common workflows. They run fast and mostly serve a canary to ensure that we are not crippling some use cases. When a bug hits us, a test case is added/modified for it. They’re mostly a reflection of the current requirement of the project.

▲

dprkh 2 hours ago | parent | prev | next [-]

I used an LLM to build this

https://github.com/dprkh/eventfs

It has good test coverage, mostly unit tests but also a number of end-to-end tests. I also made the LLM build a benchmark, which you can find at the bottom of the readme. It is obviously slow, but I thought that it is good enough to work. When I tried to write a 1 GiB file, I found that it broke down, and after writing half the file, the speed went to under one megabyte per second. Implementation is 10k+ LoC, and I have no idea what is going on there.

▲

jaggederest 2 hours ago | parent | next [-]

That's interesting because I would feed that benchmark back into the agent and loop over it, to see how much faster you could get it, and agents are really good at that kind of recursive optimization. And I would definitely add at least a simulated 1GiB write test, probably a real one honestly, if I was building something like that.

At least with agent-run tests I care about loop speed a lot, but I care about complete coverage more, so having the odd heavy weight full stack integration test is fine, I think.

▲

dprkh 2 hours ago | parent [-]

You're right. This was just a performance issue, but what if next time it is a corruption bug or a security vulnerability or really anything that can cause real consequences if happened in production? I don't think that LLM systems are inherently bound to have this flaw, but I think that we are pretty far from harnesses and algorithms becoming advanced enough so that the LLM system can kind of continuously evaluate its output and ensure it is good in all aspects.

	▲	jaggederest an hour ago \| parent [-]
		I don't know about that, Fable is, when properly guided, a better engineer for those things than I am. Narrow breadth, weird priorities, myopic and ivory tower as hell, but superhuman. Maybe that says more about me, or maybe not, but certainly it's caught bugs I would not have, and point it at things like a fuzzer, woo buddy, it has been many years since I broke out valgrind and nailed down a memory leak, but it sure can.

▲

nzeid 2 hours ago | parent | prev [-]

Yep.

▲

rapind 2 hours ago | parent | prev | next [-]

100% this is what I've done. I sucked it up and adapted myself to the tool (agents) by having as many implicit guardrails (static typing, functional, no nulls, great linting) and then layering on explicit guardrails (TDD) on top. I also want my workflow to be portable because I don't really trust the frontier model providers.

It is different though. Basically a lot of what I do has changed over the last 2 years. I totally get that a lot of people won't want to adapt though.

	▲	skydhash 2 hours ago \| parent [-]
		> I totally get that a lot of people won't want to adapt though. Or people don't want to be reverse centaur keeping the clankers happily running. Instead of helping to solve users/consumers problem.

▲

nomel an hour ago | parent | prev [-]

Yeap. Hundreds of tests for small tools are completely trivial now.

▲

stillpointlab 2 hours ago | parent | prev | next [-]

I've just been carefully reading the code. It is easy to slip into just accepting what comes out to speed things up, but reading the code is important.

I save myself by skimming things like tests, templates, some UI. Anything cosmetic. But I have to read the majority of code that ends up on my back end systems.

▲

dylan604 2 hours ago | parent | prev | next [-]

And for those that have similar-ish sentiments, what mental defect is had that prevents them from just drinking that sweet tasty kool-aid and just use the slop created. What demented trait is in them that causes everyone to just be a stick in the mud trying to ruin everyone else's good time?

In my personal experience, the ones most enthusiastic about LLM magic are those that can't code, but can now walk away with something functional if not quite the best code. Now that they can produce workable code, it will make everyone better. Yet, they have no idea how maintainable the slop is or if it's slop at all.

▲

dools 2 hours ago | parent | prev [-]

Don’t read the code!!

▲

jaggederest 2 hours ago | parent [-]

I actually dispute this, I read all the code, the core thing people have to give up is not "reading the code" per se, it's giving up on "that's not how I would have done it".

When you see a perfectly clear function or object that just isn't your style, you have to accept it and move on. Where there are concrete concerns, or it's unreadable, demand excellence, but treat it like a coworker, not an IDE.

	▲	FunHearing3443 2 hours ago \| parent [-]
		Yeah this is how I feel about it. Does it look correct? is it doing something weird? Is it forgetting about some gotcha in our domain that it hasn't been taught about yet? Otherwise, ship it.