> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.

I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead.

I can tell you right now:

1) It's 100x more than I could have achieved with zero compiler design experience.

2) I'm HIGHLY skeptical that LLMs can build something of this complexity (in some ways it's more difficult than implementing a Rust compiler) - so the testing is quite robust - 3 different systems (unit, integration, fuzz tests) each with mutant testing, each with between ~65-90% line coverage and ~50-80% branch coverage, combined with ~99% line coverage and ~86% branch coverage.

There is ZERO chance I could get something even close to this level of "working" by myself ever - let alone with minimal effort.

The test is kind of simple - if LLM's can do this... They should be able to do just about anything... Compilers are notoriously difficult to verify they actually work, rather than just kind of work sometimes...

People can say I'm wasting my time all they want.

But, one, it's been enlightening. I'm literally in awe of what they can do and have done.

Two, I've developed a bunch of tooling / metrics necessary to get them to be able to do something at this level of complexity without falling over themselves. And I think it can work at scale pretty easily.

Nearly all of the research comes from the 80s or farther back for the complexity metrics.

I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.

are we just re-inventing playwright tests except 10x slower and infinity times more expensive?

i feel like im going insane

So, throw out the traditional test pyramid, shift right, and rely more on persona testing than fine-grained atomic tests? I would hope teams don't need to re-learn that lesson for themselves, but...

This already exists. You mean capturing user flows which should already be supplied by product to the developer. A decent system is Behavior Driven Development (though honestly a poor acronym for it’s use).

>Scenario testing is the new word

How is scenario different from a behavior (as in Behavior-Driven Development)?

Gherkin and things like Cucumber are not something new, are they?

> Two of the reasons I never liked writing tests

Are you an engineer ? You must test your "creation". Or would you expect that the microwave owen you just bougth will be tested by your child while getting burned ?

Don't worry, AI maximalists have a solution: create tests for the tests.

Well you see, you just run the same test 10,000 times, and then...

I think this is just TDD or unit test dogma and I’m personally not a fan.

Unit tests and deterministic tests are hard to get right and need to be done at the correct boundary.

I have seen many people dogmatically pushing unit tests religiously but this often leads to very hard to maintain tests that mostly exist just to change along with the main code itself.

A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak.