The author does not understand what LLMs and coding tools are capable of today.

> LLMs get endlessly confused: they assume the code they wrote actually works; when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over. This is exactly the opposite of what I am looking for. Software engineers test their work as they go. When tests fail, they can check in with their mental model to decide whether to fix the code or the tests, or just to gather more data before making a decision. When they get frustrated, they can reach for help by talking things through. And although sometimes they do delete it all and start over, they do so with a clearer understanding of the problem.

My experiences are based on using Cline with Anthropic Sonnet 3.7 doing TDD on Rails, and have a very different experience. I instruct the model to write tests before any code and it does. It works in small enough chunks that I can review each one. When tests fail, it tends to reason very well about why and fixes the appropriate place. It is very common for the LLM to consult more code as it goes to learn more.

It's certainly not perfect but it works about as well, if not better, than a human junior engineer. Sometimes it can't solve a bug, but human junior engineers get in the same situation too.

It's funny always seeing comments like this. I call them "skill issue" comments.

The reality is the author very much understands what's available today. Zed, after all, is building out a lot of AI-focused features in its editor and that includes leveraging SOTA LLMs.

> It's certainly not perfect but it works about as well, if not better, than a human junior engineer. Sometimes it can't solve a bug, but human junior engineers get in the same situation too.

I wonder if comments like this are more of a reflection on how bad the hiring pool was even a few years ago than a reflection of how capable LLMs are. I would be distraught if I hired a junior eng with less wherewithal and capabilities than Sonnet 3.7.

I share examples of LLM fails on our company Slack and every week LLMs do the opposite of what I tell them.

I say capture logs without overriding console methods -> they override console methods.

YOU ARE NOT ALLOWED TO CHANGE THE TESTS -> test changed

Or they insert various sleep calls into a test to work around race conditions.

This is all from Claude Sonnet 4.

I believe that they work particularly well for CRUD in known frameworks like Rails.

OTOH I tried building a native Windows Application using Direct2D in Rust and it was a disaster.

I wish people could be a bit more open about what they build.

> The author does not understand what LLMs and coding tools are capable of today.

Claiming that the people making an AI coding tool (Zed) don't know LLM coding tools is both preposterous and extremely arrogant.

> it works about as well, if not better, than a human junior engineer.

I see this line of reasoning a lot from AI-advocates and honestly it's depressing. Do you see less experienced engineers as nothing more than outputters of code? Is the entire point of being "junior" at something that you can learn and grow, which these LLM tools cannot.

it's very well documented behavior that models try to pass failed test with hacks and tricks (hard coding solutions and so on)

From what I've experienced, this depends very much on the programming language, platform, and business domain.

I haven't tried it with Rails myself (haven't touched Ruby in years, to be honest), but it doesn't surprise me that it would work well there. Ruby on Rails programming culture is remarkably consistent about how to do things. I would guess that means that the LLM is able to derive a somewhat (for lack of a better word) saner model from its training data.

By contrast, what it does with Python can get pretty messy pretty quickly. One of the biggest problems I've had with it is that it tends to use a random hodgepodge of different Python coding idioms. That makes TDD particularly challenging because you'll get tests that are well designed for code that's engineered to follow one pattern of changes, written against a SUT that follows conventions that lead to a completely different pattern of changes. The result is horribly brittle tests that repeatedly break for spurious reasons.

And then iterating on it gets pretty wild, too. My favorite behavior is when the real defect is "oops I forgot to sort the results of the query" and the suggested solution is "rip out SqlAlchemy and replace it with Django."

R code is even worse; even getting it to produce code that follows a spec in the first place can be a challenge.

My experience so far is that, if you're limiting the "capacity" to junior engineer, yes, especially when it's seen a problem before. It's able to quickly realize a solution and confirm the solution works.

It does not works so well for any problems it has not seen before. At that point you need to explain the problem, and instruct the solution. So a that point, you're just acting as a mentor instead of using your capacity to just implement the solution yourself.

My whole team has really bought into the "claude-code" way of doing side tasks that have been on the backlog for years, think like simple refactors, or secondary analytic systems. Basically any well-trodden path that is mostly constrained by time that none of us are given, are perfect for these agents right now.

Personally I'm enjoying the ability to highlight a section of code and ask the LLM to explain this to me like I'm 5, or look for any potential race conditions. For those archiac, fragile monolithic blocks of code that stick around long after the original engineers have left, it's magical to use the LLM to wrap my head around that.

I haven't found it can write these things any better though, and that is the key here. It's not very good at creating new things that aren't commonly seen. It also has a code style that is quite different than what already exists. So when it does inject code, often times it has to be rewritten to fit the style around it. Already, I'm hearing whispers of people say things like "code written for the AI to read." That's where my eyes roll because the payoff for the extra mental bandwidth doesn't seem worth it right now.

> It works in small ... chunks

Yup.

> I ... review each one

Yup.

These two practices are core to your success. GenAI hangs reliably hangs itself given longer rope.

Agreed. This passage in particular

> when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over

Is the EXACT OPPOSITE of what LLM's tend to do. They are very stubborn in their approach and will keep at it often until you rollback to a previous prompt. Them deleting code tends to happen on command, except specifically if I do TDD, which may as well be a preemptive command to do so.

> The author does not understand what LLMs and coding tools are capable of today.

Not really, I would say they used it well and understood the limitations of LLM exactly. No matter how much polished the output or how good it is, LLMs can't build mental models of a codebase like a human does because they are just statistical machines.

in my experience TDD is a very powerful paradigm for use with LLMs.

it does a good enough job of wrangling behavior via implied context of the test-space that it seems to really reduce the amount of explanation needed and surprise garbage output.

> The author does not understand what LLMs and coding tools are capable of today.

Uh...

This author is developing "LLMs and coding tools of today." It's not like they're just making a typical CRUD Rails app.

I have had cline add ignore tags when instructed to fix type errors. That was frustrating but fixed with some stern words. I now know this failure mode and can handle it in the standard prompt (actually I expect that evetually the Cline people will).

I spent about 4 hours yesterday vibe coding a rip-off of a commercial product in a language I’ve never used, and I’m a lawyer, not a programmer.

it probably because the author uses the useless implementation of the Zed agent

it also tends to write bad tests, like a junior...