Yeeeehaaaaa, the vibes shall never end!

On a more serious note, they were mostly f*cked by their paas provider imo. Claude will always do dumb shit. Especially if you tell it to not do something... By doing so you generally increase the likelihood of it doing it.

It's even obvious why if you think about it, the pattern of "you had one job, but you failed" or "only this can't happen, it happened!" And all it's other forms is all over literature, online content etc.

But their PaaS provider not scoping permissions properly is the root cause, all things considered. While Claude did cause this issue there, something else would've happened eventually otherwise.

The problem is that destruction isn't contained to the company. If an AI agent exposes all company data and that includes PII or health information, that could have an impact on a large number of people.

Normalisation of deviance is the problem: https://en.wikipedia.org/wiki/Normalization_of_deviance

Remember that these models are getting better; this means they get trusted with increasingly more important things by the time an error explodes in someone's face.

It would be very bad if the thing which explodes is something you value which was handed off to an AI by someone who incorrectly thought it safe.

AI companies which don't openly report that their AI can make mistakes are being dishonest, and that dishonesty would make this normalization of deviance even more prevelant than it already is.

Yep. I don’t see how that metric indicates how… strong(?) a language model is.

> That is like telling students to never do a PhD thesis on superscalar out-of-order execution, stochastic gradient descent, or UDP.

No decent PhD advisor would let their PhD student base their PhD thesis on such well-known concepts: a doctoral study programme is a journey into something never-seen-before (with a very high likelihood of faling and shattering your life). Anything else is failure.

(Obvious exception: either he or the PhD student can convince the other one that there could be something really, really deep in, say, "superscalar out-of-order execution", "stochastic gradient descent" or UDP be found that generations of researchers overlooked, and which once discovered might necessitate rewriting all the standard textbooks about this topic).

Yea, but they'll do the job. What else plausibly could? ...an LLM? Then you're back at unreliable computation.

That’s not a particular reading

You aren't alone.

Not even a few years ago if you introduced a component to a system that would result in non-deterministic output... Hell, a single function... You would be named and shamed for it because it went against every principle you should be learning as a novice writer of software.

I have used the LLM tools, and I see the real-world potential for these things. But how it's all being sold and applied now: it's upside down.

Replace fluoride with lead in the water. Blocks out all the negative effects from wind turbines. /s

With the old way of doing things you could spend energy to reduce errors, and balance that against the entropy of you environment/new features/whatever at a rate appropriate for your problem.

It's not obvious if that's the case with llm based development. Of course you could 'use llms until things get crazy then stop' but that doesn't seem part of the zeitgeist.

> It has always been non-deterministic but we relied on low level engineers who knew the dark magicks to keep the horrors at bay.

This is a disingenuous comparison.

First of all, what you're talking about is nondeterminism at the hardware level, subverting the software, which is, on an ideal/theoretical computer, fully deterministic (except in ways that we specifically tell it not to be, through the use of PRNGs or real entropy sources).

Second of all, the frequency with which traditional programs are nondeterministic in this manner is multiple orders of magnitude less than the frequency of nondeterminism in LLMs. (Frankly, I'd put that latter number at 1.)

This is part of a class of bullshit and weaselly replies that I've seen attempting to defend LLMs over the years, where the LLMs' fundamental characteristics are downplayed because whatever they're being compared to occasionally exhibits some similar behavior—regardless of the fact that it's less frequent, more predictable, and more easily mitigated.

You misunderstand.

The only reason to launch subagents is to avoid poisoning the LLM's already small context window with unrelated tokens.

It doesn't make the LLM smarter or more capable.

Wtf? A sub-agent is a tool you give an agent and say "If you need to analyze logs delegate to the logs_viewer agent" so that the context window doesn't fill up with hundreds of thousands of tokens unnecessarily. In what universe do you live in where that mechanism somehow means you need fewer humans?

Do you think this means "Build a car" can be accomplished just because an LLM can send a prompt to another LLM who reports back a response?

Claude does not have a "theory" of anything, and I'd argue applying that mental model to LLM+Tools is a major reason why Claude can delete a production database.

So... systemd is AGI now?

Most of us work on software that can modify its own configuration and create new jobs. I, too, have worked in ansible and terraform.

The key break here is the lack of predictability and I think it's important that we don't get too starry eyed and accept that that might be a weakness - not a strength.