Remix.run Logo
▲ gzell 5 hours ago

From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don't wag the dog.

▲donkey_brains 4 hours ago | parent | next [-]

Very true. Which makes the conclusion reached be this study even worse: we are stuck reviewing endless PRs that solve the wrong problems.

▲hibikir an hour ago | parent [-]

We were there many years ago too though: I had to take over a rather complicated system circa 2012, which had been designed to be built by a team of 20 or so, for timelines, but they had little effort on integration. This meant that a whole lot of guards were written all over the place for situations that, in practice, never occurred. To try to get this to production in a timely fashion, I ended up putting metrics per guard, along with metrics for what was caught by broad error checks near the base of the program. Basically all the defensive programming that had been built was pointless, while I had to write a whole lot of cases that were being dropped, and the people that designed things up front never considered.

AI error handling in code review is a lot like said defensive programming in a vacuum: It'll take a lot of code and a lot of energy, but it's unlikely to be important. Even when you are designing something that gets executed hundreds of millions of times a minute, over-review is easy. You pay for it when letting errors in the real world are unacceptable, but it's pretty inefficient. And that's without getting into features that just don't get used, so the entire thing should not have been built in the first place.

▲diego_moita 4 hours ago | parent | prev | next [-]

I don't fully disagree with you. That is a real big problem.

But both things can be true. One problem doesn't exclude the other.

▲wilkystyle 3 hours ago | parent | prev | next [-]

With all respect to your personal experience, I could not possibly disagree more strongly with this.

In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. "No Plan Survives First Contact With the Enemy".

The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.

Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.

IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.

You can of course mitigate this by staying more closely involved with the work as it's being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.

A sister comment mentioned a plethora of PRs that don't solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn't foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.

▲blurbleblurble 2 hours ago | parent [-]

I don't think they're talking about "the perfect specification" they're talking about correctly describing what the program should do at all, formally or informally.

▲mathisfun123 4 hours ago | parent | prev | next [-]

> The tail don't wag the dog

If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.

▲skydhash 4 hours ago | parent [-]

You may need to check the pipeline so that the refinery isn't stuck filtering random stuff from the oil.

AKA, it would be nice to have nice patches like this one to review.

https://cgit.freebsd.org/src/commit/?id=09d62ba33de113d01e88...

▲mathisfun123 3 hours ago | parent [-]

you're avoiding the question: If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say i was wrong?

▲skydhash 3 hours ago | parent [-]

I’m not avoiding. But you have to state your assumption that everything before the pipeline is working well and it’s just the refinery that is struggling.

Then your analogy would break down as we all know that the PR machine is a slop producer.

▲webdood90 3 hours ago | parent | prev [-]

So the solution is: write a perfect spec and then you'll have perfect code, no need for reviews?

What world do y'all live in where this is your reality?

▲donkey_brains 3 hours ago | parent [-]

Here’s the thing: a human engineer is embedded in their company. They sit in meetings, they know whose opinions are more important and whose they can disregard. They know the history of how people at the company like to do things. They use all of this context to feed more precise specifications into a coding agent, and as a result, they get an output that better suits the environment of their company. Ergo, they spend less time reviewing agent output which was built to solve the wrong problems in the wrong way, and which ultimately must be discarded or significantly rewritten.