> For example, in testing by the investment firm Millennium, Fable 5.1 found the cause of a rare crash on their internal systems that none of their engineers (or any other model) had been able to explain after several years of trying.

Say what you will about LLM-generated code, but stories like this give me hope that software will never be as buggy as it once was.

Software will be buggier than ever but also way less buggy.

The marketing here trick is, if they spent the same money on humans they'd have found it years ago.

Instead, the lurking variable here is new budget was added. With the new budget, they added a new tool, and the bug was located.

The difference here was budget.

That kind of one shot capability is impressive but how does it work for my typical work style? The way I work is to build a huge roadmap with goals and hand it to my agent to execute (often over night). I don't care that much about the benchmarks, what I care about is how often Fable 5.1 is making a baffling decision and destroys my plan, not respecting stop conditions or goals. I would seek for behavioral reliability over long autonomous runs, not eval scores. Anyone have that kind of feedback and observations?

My experience with such problems, is that they stay for this long because nobody cares, not because it's impossible, or even technically too difficult. So hopefully, LLM will improve things, but that quote is a lie.

I think we'll have lots of bugs. They'll just be found and closed way sooner. You'll have an agent that watchs for issues, then opens a PR fixing it.

If only!

I sometimes have that feeling too, then ask another LLM to do a code and vulnerability review and OMG: rookie mistakes, over complications and security gaps even a 1st year student would not make regularly.

So.. one more year of untreated bipolar AI psychosis I guess..

> Say what you will about LLM-generated code, but stories like this give me hope that software will never be as buggy as it once was.

Stories like these is what I now call 'Marketing slop'

We will have more bugs. Even the best models with the best software engineers will produce bugs. There are two reasons : first the pressure to produce more and second LLMs will always produce slop

[flagged]

You have a serious engineering problem if you're not able to find the source of a crash after years.