Remix.run Logo
danbruc 3 days ago

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.

Supermancho 3 days ago | parent | next [-]

> Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability

My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:

* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.

* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.

* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.

* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.

* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.

bla3 3 days ago | parent [-]

This sounds roughly right to me, except for "maintainability". In my experience, agents really don't like deleting code unless you explicitly ask for it. If you're not careful, you end up with new better implementations of things but with the old implementation still around in perpetuity. Humans do this too of course.

mwigdahl 3 days ago | parent | next [-]

They're being trained by a lot of repetitions of one-shot bugfixes and feature implementations. If they don't have to maintain code over the long term they have no incentive to not take the easy win with lower immediate risk and higher longer-term maintenance burden.

Supermancho 3 days ago | parent | prev | next [-]

The shadow of Chesterton's Fence is relevant. The LLM doesn't know why code exists, if it doesnt start traversing up the scope of a project. Even then, it can't be sure that the code isn't a dependency of something else outside the project (especially if there's a side effect). I'm not sure it's ever going to be easy to address this concern in a straightforward and portable way.

I do sometimes see duplicate functions, which is troubling.

ACCount37 3 days ago | parent | prev [-]

Humans had to get it drilled into them that "+12 -440" is a damn good line stat, and that keeping around dead code is bad, especially in the age of version control.

Not too surprised that LLMs also don't "get it" by default?

danielvaughn 3 days ago | parent | prev | next [-]

Yes. I use these models day in and day out, on all sorts of tasks. I cannot believe I hear people say that coding is solved.

dnikolovv 3 days ago | parent | next [-]

Right. It feels like we're living in a parallel world or something. Can agents code? Sure. Can you let them code on their own for a serious production project? Not a chance.

heaney-555 3 days ago | parent [-]

Plenty of serious production projects are doing exactly that. Are you using GPT-6 Astra, or something older?

an0malous 3 days ago | parent | next [-]

Which serious production projects have AI agents coding on their own? And I’m assuming that means they are routinely taking tasks and deploying them to production autonomously

margalabargala 3 days ago | parent | prev | next [-]

You and the person you are replying to are talking about different things.

Agents cannot be given a high level goal and then left unsupervised, for hours, without making some dumb decisions.

denverllc 3 days ago | parent | prev | next [-]

Astra does exactly the same sort of things that Sol or any of the previous agents do. They duplicate code, overengineer, miss the point, etc.

I was very optimistic about it when it was announced and saw all the demos, but a week later I find it only marginally better (and in some cases worse) than before.

Mawr 2 days ago | parent | prev [-]

Yeah, Astra, not like it matters. It will gleefully make db queries that perform full table scans instead of adding an index. I need to intuit that that's what's happening and then explicitly instruct it not to be negligently dumb.

LevGoldstein 3 days ago | parent | prev | next [-]

Consider that the people who have never have never had to be on-call for a product that results in losses for the company in the event of an outage, and/or have never had to attend an RCA meeting have just as much of a voice in this discussion. Some segments of the software industry don't even have to worry about those sorts of things, and it's easy to get myopic after a while...I doubt that a long time dev at hobbyist-centric wearable manufacturers or audio plugin shops has the same sorts of concerns as someone who has specialized in ad delivery for social media sites.

jimbokun 3 days ago | parent [-]

Allowing only people who have served on call or made concrete contributions to multiple RCAs to have input on the software development process is a wonderful idea!

softwarewright 3 days ago | parent | prev [-]

me too; and my coding agents are slowed down (from developing features) because I require them to refactor the code to be more readable; my code metrics tests force AIs to leverage functional programming and design patterns).

Yes agents can produce code that compiles and runs, but I had to add tools to keep them on track, document their work, follow a process, check their outputs. I also use other AIs to generate developer documentation and review code.

It is like managing a bunch of idiot savant eager-to-please interns, except unlike interns, coding agents do not (yet) learn and improve on their own.

chucksmash 3 days ago | parent | prev | next [-]

Correctness is not a binary thing though.

I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?

danbruc 3 days ago | parent [-]

Sure, every sufficiently large codebase will have bugs somewhere, but it will work correctly at least something like 99.999 % of the time after ironing out the bugs on the common code paths. But that does not change the fact that being [mostly] correct is the lowest bar you have to cross.

einrealist 3 days ago | parent | prev | next [-]

And there is another problem: LLMs generating too much code, code that is doing more than was asked. And that cannot be fixed by tests. Usually, we create tests for wanted behavior and expected exceptions. But we don't create tests for undesired behavior.

flyinglizard 3 days ago | parent | next [-]

Code cost is almost down to zero. If you move the point of “just leave it to the machine” from the compiler (where humans used to do the coding) to the high level logic (now with LLMs) then in most cases more code does not really matter. Like, why build and maintain an abstraction where the LLM could implement this many times over each time with different subtleties? Why use a library with its own constraints when you could have exactly what you want? Why use cross platform frameworks when you can just one shot the thing to N different platforms? It’s not even slower. You can have code that’s larger yet more performant (stripping away abstractions can do that).

From time to time I try to do a pass of coalescing flows and cases and removing dead code to reduce the context and prevent the LLM from tripping over itself. But if it’s exclusively LLM maintained code I don’t care too much if there’s more of it.

pdimitar 3 days ago | parent [-]

The "exactly" part is the problem.

Just last two weeks I had to slap Fable, three times, to stop writing 1000-2000 lines of defensive code... because of DB columns I just forgot should be NOT NULL. That was it. Nothing else. I told it that, boom, -4800 coding lines: gone.

LLMs defend the status quo and they regularly lose sight of everything bigger than the current PR they are working on.

I too am gradually making peace with the fact that LLM-maintained code does not have to be 100% readable for humans.

But this is not about readability. It's about the data model. So one concession I am willing to make is: don't care too much about the code _BUT_ manually curate the data model. So far: small wins on iteration turns and code volume producing. Too early to tell but for now I am happy with the results.

user43928 3 days ago | parent | prev [-]

Have you worked with Opus 5?

Its documentation about what the code does not do could fill whole books.

UI copy being full of slop explaining what the software does not do is another problem.

I am not convinced that a lack of negative test cases is an issue.

I do agree it generates too much code most of the time.

lukeschlather 3 days ago | parent [-]

Documentation is important. I would say Opus' propensity to write documentation that documents non-features is part of the problem being discussed.

And the problem isn't just that it says what the software doesn't do, most of the things it claims are in fact meaningless, it's not even clearly describing something the software shouldn't do.

gedy 3 days ago | parent | prev | next [-]

"Coding" is just a poor term for this, as there's so much room to weasel different meanings out of it. At every company I've worked in past 20+ years the "coders" were engineering a product from wrong or unclear requirements and specs from non-technical people. The act of coding was secondary (but an important throttling function to make us stop and think about what is even possible or makes sense.) Never did FAANGs, so YMMV.

Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.

perchard 3 days ago | parent | prev | next [-]

second sentence of the article: "Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC)."

glenstein 3 days ago | parent | prev | next [-]

I think this is a two things can be true situation, where our colloquial meaning of coding is not quite adequate to indicate the full range of criteria that really matter, but also, that criteria as a totality (maintainability, reliability etc) is something that can also be targeted and optimized for.

This reminds me a bit of a PhD Comics webcomic that confidently claimed we would "never" cure cancer, on the grounds that cancer is not one thing. And I don't know that we will ever actually cure cancer, but that wouldn't be the reason. Correctly noting the problem space is bigger than a layperson would initially appreciate is a lot of things, most of them helpful, but the one thing it's not is a formal a demonstration of optimizing against the problem space as a whole.

3 days ago | parent | prev | next [-]
[deleted]
gchamonlive 3 days ago | parent | prev | next [-]

It's not solved, I agree. But if we pretend it is we can prepare for when it actually becomes solved, if ever, and measuring sloppiness is a worthy pursuit even if we never "solve code".

binary0010 3 days ago | parent | prev | next [-]

I just setup a large refactor with Astra and was feeling super lazy and let it mostly do it without my usual extreme micro managing.

The refactor ended up adding 22,000 loc.

I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.

In case you are wondering why vibe coders are doing 30k loc a day, this is why.

ryandrake 3 days ago | parent | prev | next [-]

This has been a problem in the software industry for a loooong time, though. Too many developers see "It works" as the end state, a signal that you're done. "It works" is just step 1.

3 days ago | parent | prev | next [-]
[deleted]
anon-3988 3 days ago | parent | prev | next [-]

Look, we are comparing against the average developer here And yes, they are obsolete

mr_roboto 3 days ago | parent | prev | next [-]

You come across as someone who has never worked on a real software project. Humans create tons of bugs on a regular basis. AI is already better than most programmers.

whatever1 3 days ago | parent | next [-]

Better at writing one piece of code, maybe.

Better at writing code within a huge system, definitely not. Maybe in the future, but as of Astra, Fable 5.1, the answer is still no.

econ 3 days ago | parent | next [-]

I'm definitely not experienced enough to know but I read people are having it write somewhat elaborate documentation beforehand. Have it figure out which parts of the code may be touched, what other things will be affected and which uncertainties it has. Basically a full report before giving permission for any code to be written/modified. Again, I haven't seen it, don't know how common this is nor how effective. Though it sounded interesting.

user43928 3 days ago | parent | prev [-]

Disagree. I have 200k LOC now plus 100k in tests, and it is still performing like it was four months ago when I started to seriously use AI.

If anything, it works more reliably today with the smarter models.

danbruc 3 days ago | parent | prev | next [-]

I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.

mr_roboto 3 days ago | parent [-]

I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've seen much worse from teams of humans and I've accepted the drawbacks that are slowly going away with each new model.

bigstrat2003 3 days ago | parent | next [-]

> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.

That is never worth it. You're ruining the software you work on when you do this.

batshit_beaver 3 days ago | parent | prev [-]

> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.

The speedup of slop production being “worth it” is what we, as a society, are having trouble evaluating at this point in time. In all likelihood it’s worth it only in the short term.

OtomotO 3 days ago | parent | prev [-]

AI is the average of all programmers.

It's just that many (I guess that includes me? :D) assumed that they are better than the actually were.

ipsod 3 days ago | parent [-]

A team of programmers is "worse" than their best member. Worse quality of code, worse thinking. You can usually get more done with a lot of mediocre hours of work than a few brilliant hours, though.

AI I've used isn't a better coder than I am - it's just got a lot more hours in an hour than I do.

_s_a_m_ 3 days ago | parent | prev | next [-]

Amen. I sometimes wonder if all programmers are now marketing people who know shit about software development and engineering.

zsoltkacsandi 3 days ago | parent | prev | next [-]

That is very well put and summarizes what distinguishes real software development from vibe coding.

ahalay-mahalay 3 days ago | parent | next [-]

I’m pretty sure that same discourse was seen every mass production epoch, from textiles to electronics. Yet here we are, hand-crafted high quality things are rare and expensive.

Terr_ 3 days ago | parent [-]

> mass production epoch, from textiles to electronics

I see this comparison a lot, and I think it's a trap, because it invites us to confuse scaling duplicates with scaling design changes.

Duplicative mass-production was always core to software from the moment it first became "soft". A factory churning out 10,000 copies of the same book maps to 10,000 downloads of a single software release. The paper and bindings of the book may be below hand-crafted standards, but the words are largely unaffected.

In contrast, LLM-coding is the design and prototyping stage. So if we want to learn from textiles/electronics, we shouldn't be thinking of acres of looms, but instead about fashion-design, custom tailoring, determining patterns for clothes, designing new appliances, choosing circuit layouts, etc.

Dlemlo 3 days ago | parent | prev [-]

I have seen so much production code with garbage code and massive bugs, the industry doesn't care for 'real'.

danbruc 3 days ago | parent [-]

They care in principle, for the most part bad code makes changes and extensions slow to implement and causes unnecessary production issues which costs time and money. But there is always the tension between implementing something quickly now and being able to implement things quickly in the future and unfortunately the preference is almost always quickly now despite everyone knowing that this is the way more expensive choice in the long run.

hax0ron3 3 days ago | parent | next [-]

It isn't necessarily the more expensive choice in the long run. Let's say that companies A and B are direct competitors who start from the same point. Company A quickly codes a bunch of buggy software and ships it in a month. Company B takes its time and ships good, clean, well-organized, mostly bug-free code in five months. Company A makes money. Company B goes out of business and its code is useless, its nice qualities irrelevant.

This is just a hypothetical example, I'm not saying that this is how it would necessarily go in all cases.

Dlemlo 3 days ago | parent | prev [-]

Not even in principle. Like how often you had to fight a product manager to do this or that.

"Does that code work?" "yes" "so lets ship it" "but its not good" "but it works right?"

bigstrat2003 3 days ago | parent | prev | next [-]

Yeah, the author leads with a blatantly false claim that models are almost perfect at generating code. They are not. They write pretty bad code, worse than any capable human would.

echelon 3 days ago | parent | prev [-]

And all of these things will be solved one by one.

It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.

Why do you not think these things aren't going to be completely automated? What makes these tasks special?

Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.

Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.

Nano Banana and GPT Image can do a better job than graphics designers.

LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.

Just wait. All of these things will be solved.

There is no "stopping point".

Edit:

Don't anticipate that 2036 will look anything like 2026.

Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.

sigbottle 3 days ago | parent | next [-]

May the iterative loop of adding new axes to evaluate on be a natural, healthy progression, instead of needing to frame it as an us-them problem?

If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.

If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.

mbernstein 3 days ago | parent | prev | next [-]

I think you need to define what solved means and what better means.

Have you tried one-shotting real distributed systems problems? What was the result and how did you verify correctness?

Dlemlo 3 days ago | parent [-]

Like the 0.1% we do is your counter example?

But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.

I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.

bigstrat2003 3 days ago | parent | prev | next [-]

> It's astounding to me that people can see coding get solved...

We haven't seen that. Maybe when we do, we will start to believe that other things will get solved.

danbruc 3 days ago | parent | prev | next [-]

I have no doubt that AI will eventually be able to write essentially perfect code, I am just saying that we are still quite far away from that point.

BowBun 3 days ago | parent | prev | next [-]

> compelling novel game loops

Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).

dwroberts 3 days ago | parent | prev | next [-]

I think the usage of solved is silly. Things get automated, they basically never get ‘solved’.

Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.

testaccount121 3 days ago | parent | prev | next [-]

..

echelon 3 days ago | parent [-]

People are ego-centric and pull a blindfold over their eyes.

claudeslop 3 days ago | parent | prev | next [-]

[dead]

_s_a_m_ 3 days ago | parent | prev [-]

[flagged]

dang 3 days ago | parent | next [-]

Personal attacks will get you banned here. No more of this, please, regardless of how wrong someone is or you feel they are.

https://news.ycombinator.com/newsguidelines.html

sceptic123 3 days ago | parent | prev [-]

[flagged]