There may be an "LLM Winter" as people discover that LLMs can't be trusted to do anything. Look for frantic efforts by companies to offload responsibility for LLM mistakes onto consumers. We've got to have something that has solid "I don't know" and "I don't know how to do this" outputs. We're starting to see reports of LLM usage having negative value for programmers, even though they think it's helping. Too much effort goes into cleaning up LLM messes.

> Look for frantic efforts by companies to offload responsibility for LLM mistakes onto consumers.

Not just by companies. We see this from enthusiastic consumers as well, on this very forum. Or it might just be astroturfing, it's hard to tell.

The mantra is that in order to extract value from LLMs, the user must have a certain level of knowledge and skill of how to use them. "Prompt engineering", now reframed as "context engineering", has become this practice that separates anyone who feels these tools are wasting their time more than they're helping, and those who feel that it's making them many times more productive. The tools themselves are never the issue. Clearly it's the user who lacks skill.

This narrative permeates blog posts and discussion forums. It was recently reinforced by a misinterpretation of a METR study.

To be clear: using any tool to its full potential does require a certain skill level. What I'm objecting to is the blanket statement that people who don't find LLMs to be a net benefit to their workflow lack the skills to do so. This is insulting to smart and capable engineers with many years of experience working with software. LLMs are not this alien technology that require a degree to use correctly. Understanding how they work, feeding them the right context, and being familiar with the related tools and concepts, does not require an engineering specialization. Anyone claiming it does is trying to sell you something; either LLMs themselves, or the idea that they're more capable than those criticizing this technology.

People can't be trusted to do anything either, which is why we have guardrails and checks and balances and audits. That is why in software, for instance, we have code reviews and tests and monitoring and other best practices. That is probably also why LLMs have made the most headway in software development; we already know how to deal with unreliable workers that are humans and we can simply transfer that knowledge over.

As was discussed on a subthread on HN a few weeks ago, the key to developing successful LLM applications is going to be figuring out how to put in the necessary business-specific guardrails with a fallback to a human-in-the-loop.

We need to put the LLMs inside systems that ensure they can only do correct things.

Put an LLM on documentation or man pages. Tell the LLM to output a range of lines, and the system actually looks up those lines and quotes them. The overall effect is that the LLM can do some free-form output, but is expected to provide a citation to support its claims; and the citation can't be hallucinated, since the LLM doesn't generate the citation, a plain old computer program does.

And we haven't seen LLMs integrated with type systems yet. There are very powerful type systems, like dependent types, that can prove things like "this function returns a list of sorted number", and the type system ensures that is ALWAYS true [0], at compile time. You have to write a lot of proof code to help the compiler do these checks at compile time, but if a LLM can write those proofs, we can trust they are correct, because only correct proofs will compile.

[0]: Or rather, almost always true. There's always the possibility of running out of memory or the power goes out.

Yeah, I can't wait for this slop generation hype circlejerk to end either. But in terms of being used by people who don't care about quality, like scammers, spammers, blogspam grifters, people trying to affect elections by poisoning the narrative, people shitting out crappy phone apps, videos, music, "art" to grift some ad revenue, gen AI is already the perfect product. Once the people who do care wake up and realise gen AI is basically useless to them, the internet will already be dead, we'll be in a post-truth, post-art, post-skill, post-democracy world and the only people whose lives will have meaningfully improved are some billionaires in california who added some billions to their net worth.

It's so depressing to watch so many smart people spend their considerable talents on the generation of utter garbage and the erosion of the social fabric of society.