The most clever lines of code are the ones you don’t write. Often this is a matter of properly defining the problem in terms of data structure. LLMs are not at all good at seeing that a data structure is inside out and that by turning it right side in, we can fix half the problems.

More significantly though, OP seems right on to me. The basic functionality of LLMs is handy for a code writing assistant, but does not replace a software engineer, and is not ever likely too no matter how many janky accessories we bolt on. LLMs are fundamentally semantic pattern matching engines, and are only problem solvers in the context of problems that are either explicitly or implicitly defined and solved in their training data. They will always require supervision because there is fundamentally no difference between a useful LLM output and a “hallucination” except the utility rating that a human judge applies to the output.

LLMs are good at solving fully defined, fully solved problems. A lot of work falls into that category, but some does not.

>> The most clever lines of code are the ones you don’t write.

Just to add, I think there are three things that LLMs don't address here, but maybe it's because they're not being asked the broader questions:

1. What are some reasonable out-of-band alternatives to coding the thing I'm being asked to code?

2. What kind of future modifications might the client want, and how can we ensure this mod will accommodate those without creating too many new constraints, but also without over-preparing for something that night not happen?

3. What is the client missing that we're also missing? This could be as simple as forgetting that under some circumstances, the same icon is being used in a UI to mean something else. Or that an error box might obscure the important thing that just triggered the error. Or that six years ago, we created a special user level called "-1" that is a reserved level for employees in training, and users on that level can't write to certain tables. And asking the question whether we want them to be able to train on the new feature, and if so, whether there are exceptions to that which would open the permissions on the DB but restrict some operations in the middleware.

"What are we missing" is 95% of my job, and unit tests are useless if you don't know all the potential valid or invalid inputs.