LLMs are the first massively popular type of computer programs which actively trying to break half a century worth of human training, which basically distills to "computer programs are highly deterministic and if they work, they are outputting correct predictable results every time" (I'm talking about average population subconscious opinion here, no need to list exceptions). Average person still can't comprehend how LLM output is random all the time and how the identical query to the same program version will produce variable results again and again.

I wonder what will happen after our benevolent prophets St.Sam and St.Dario will succeed in re-training humanity and break this collective expectation of program correctness. I guess they didn't even think about that.

> LLMs will listen to you and follow your instructions and restrictions most of the time, which seems to be enough for people to believe that they will every time.

It's called automation bias. If something works 90% of the time the human mind will extrapolate that to be 100%. That's just how humans work.

https://en.wikipedia.org/wiki/Automation_bias

Are we all heading in that direction?

I know it may seem like that reading HN but LLMs are not necessary for writing software, they might be a useful adjunct to it, but they do not have to be central to it (and Id argue they shouldn’t be).

We don’t have to head in this direction of using LLMs for most development at all.

There’s an aspect of extrapolation in the perception spike of the Dunning–Kruger effect.

In the same way smart people, doctors etc, can be better victims for scams I think tech skills can really give the wrong impression of how transformers and LLMs work. If someone has decades of relational database experience all their assumptions will be coloured towards data existing in the model accessible in a rational manner.