Remix.run Logo
▲ xyzsparetimexyz 7 hours ago

If you don't look at the code, how do you know it's higher quality than ever?

▲XCSme 7 hours ago | parent | next [-]

I simply test the final result,manual tests, automated tests, benchmarks, people using the apps and giving feedback, increased number of sales, etc.

▲mzhaase 7 hours ago | parent [-]

Here is something I found this week: copies the entire db to memory, inserts a single row into in-memory db, deletes everything from real db, writes everything back. Sentence that caused this is "do XXX in one transaction".

All tests pass, everything seemed to work. Would not have noticed of I didn't read through the code. Amount of other stupid shit I found is crazy.

▲ricardobeat 7 hours ago | parent | next [-]

These anecdotes mean little without knowing context, model, harness, prompt. The technology has moved a lot in the past year, and does require some adjustment.

My very first experience with OpenCode led to a `rm -rf /` within minutes, which I caught by chance watching the output stream in. Yet 99% of my code is currently written through an agent.

▲xigoi 5 hours ago | parent [-]

Ah yes, the favorite response to any criticism of LLMs – you are using the wrong model.

▲XCSme 7 hours ago | parent | prev [-]

A performance benchmark would have easily caught this though, ask the LLM to create and run benchmarks and see how those change after each new feature/major change.

▲verdverm 7 hours ago | parent [-]

I have found test coverage reports to be really helpful as well.

▲XCSme 7 hours ago | parent | prev | next [-]

Also to make it clear, I am not saying the code itself is better, but the end-product and what the user sees and gets. Like that saying "if it's stupid, but it works, then it's not stupid"

▲sixtram 6 hours ago | parent [-]

for you, "if it's stupid, but it works, then it's not stupid."

for me, if it's stupid, but it works, then it's a coincidence.

▲XCSme 5 hours ago | parent [-]

I think LLMs themselves are a good example of "it's stupid but it works"

▲Someone1234 7 hours ago | parent | prev | next [-]

Because the exact same processes that create it faster, are also now generating large comprehensive test suites, and data-validation suites based around the actual business requirements and data/edge cases.

In essence, it is writing the code faster (great), but then also writing the code that was often previously never written due to budget/time limitations (comprehensive testing/validation).

So, how do you know the code is the highest quality ever? Because we have test result data that PROVES it to be the case. It isn't "trust but verify" it is "Don't trust. Verify."

▲springtimesun 6 hours ago | parent | next [-]

This is my observation as well. I thought I was good at testing (I wanted to be good at testing), but the models are just better at it. Testing in particular rewards a tireless cataloging of the tiny and the tedious and in this respect LLMs are unmatched.

Single data point, my Sentry notifications across more than a half dozen client projects when from 2-4 a month to basically 0. That's while shipping 20x _more_ code.

▲mzhaase 7 hours ago | parent | prev | next [-]

Yeah but you don't test for stuff you don't know you have to test for. All tests can pass and internally its still done in a braindead way that will explode at some point.

▲Someone1234 6 hours ago | parent [-]

Absolutely, and this is exactly why humans are weaker than LLMs. We consider what is likely, they consider what the data/logic shows is possible. So you get edge cases tests/flagged, that wouldn't even be something a real person would know to look for.

▲bakugo 7 hours ago | parent | prev | next [-]

How do you know those large comprehensive test and validation suites you've never looked at actually reflect real user experiences at all?

▲Someone1234 6 hours ago | parent [-]

You must have replied to the wrong comment, since the one you did replied to never claimed that the test and validation suites were "never looked at." They are reviewed, and the test/data results are reviewed.

It said verbatim: "Don't trust. Verify."

▲verdverm 7 hours ago | parent | prev [-]

> Because we have test result data that PROVES it to be the case.

pgrust thought this too, but then another human discovered claude had hardcoded benchmark query plans.

Now pgrust has a "test" [1] to make sure the Ai doesn't do this again (unless it decides this test is bad for the current goal)

I have a growing list of these, another notable one is OpenCode having a security regression [2] because the Ai redesigned configuration loading for v2.

Real verification seems beyond transformer capabilities.

[1] https://github.com/malisper/pgrust/blob/38634b408dea373f3944...

[2] https://github.com/anomalyco/opencode/pull/54921

▲weego 7 hours ago | parent | prev | next [-]

Very little code written in the world day to day ever has to justify its 'quality' or long term value. A lot it is just one off tools, bits and bobs to make a teams life easier or whatever.

I used to work in Web agency for sizeable brands and not a single site I built lasted over a year because the companies changed concept regularly. The next agency would come in and tell them it was built wrong and couldn't be reused etc etc. We did the same, it was irrelevant how it was built and we didn't even look. This is the nature of most code. The problem is that very few people will ever be in a position to be writing verifiable important code, but every developer likes to cosplay that they are.

Most code is ultimately an, annoyance businesses need to meet a goal that will always be a cost. It's often worth treating it as such and writing things that are easy to spin up and teardown.

The people most vexed by AI dev appear to be the ones who think writing code is a pinnacle role, it's not even though over our lives the salaries would sometime suggest otherwise. That's ultimately going to be a short term blip in the landscape and most code will be one seen as the cost it is.

▲noosphr 7 hours ago | parent | prev | next [-]

A programmer's output was never code. It's was just what we had to do to get computers to work for us. Now I just ask a genie to do what I did and check the results.

▲xigoi 5 hours ago | parent [-]

Good for you. I prefer to communicate with computers using an unambiguous language that describes exactly what I want them to do.

▲noosphr an hour ago | parent [-]

Ah, a connoisseur of machine code.

▲jstanley 7 hours ago | parent | prev | next [-]

How do you know it's high quality if you do look at the code?

▲eulgro 7 hours ago | parent | prev | next [-]

Competitive programmers don't usually produce quality code, I'm guessing it could hardly be worse even without looking at it.

▲sabas123 7 hours ago | parent | next [-]

They do produce code optimizing for a different goal. But readability is often not one of them yeah.

▲XCSme 7 hours ago | parent | prev [-]

That's true, code is usually shit but the result/outcome really good and performant.

▲shmoil 7 hours ago | parent | prev | next [-]

He must have been a really bad coder.

▲rvz 7 hours ago | parent | prev [-]

Claude told him so.