I feel like every time this conversation comes up now someone has to remind everyone that an LLM is just a mathematical model. An LLM can't do anything except produce a stream of output tokens. The problems we keep seeing are tools that interpret those output tokens as actionable instructions without an adequate framework and safeguards for how they operate.

Data from LLMs being processed by these tools should be treated the same as any external input into any software system: parse - don't validate - to convert to a systematic representation with deterministic consequences and then consider those consequences within a clearly defined and limited framework. You never trust data from external sources verbatim. And you never try to use vague human language when you need to describe precise technical details unambiguously.

We learned these lessons a very long time ago in programming. It's why we have programming languages in the first place among countless other examples. But way too many people are so infatuated with LLMs and agents that they've already forgotten the basic principles of their craft after only a few months.

▲

thewhitetulip a day ago | parent [-]

> LLM is just a mathematical model

Yes those of us who bothered to know the internals know of this. But the marketing says that these are magic tools.. So that's gotta be a shock for them, but the joke is the people who irresponsibly use this won't ever read this!

▲

Silhouette a day ago | parent [-]

The thing I find concerning lately is that even a lot of technical people seem to be jumping on the hype train this time around. Obviously LLMs have become very useful tools for assisting some technical tasks but even SOTA models are nowhere near reliable and predictable enough to trust their output completely as YOLO mode agentic workflows effectively do.

Given the nature of LLMs I don't think they can ever clear that bar without some other element being introduced. The nondeterminism and chaotic nature of LLM output is enough to rule them out as a reasonable foundation for any fully automated system that would be controlling anything potentially dangerous or damaging.

But it seems to be heresy at the moment to even suggest that the future might not be bright if everyone just relies on agents driving LLMs to do all the real work. The number of people I've encountered in the past year who I'm fairly sure are smart and technically capable and yet who are also now happy to do development and other tasks either without any human in the loop at all or with at best a cursory LGTM level review before approving the LLM's output is remarkable.

	▲	thewhitetulip 21 hours ago \| parent \| next [-]
		Agreed. I see people using LLM to reply to slack or to commit a git branch!! Execs expecting to double your workload just because you have LLMs. It's insane at this point
	▲	lern_too_spel a day ago \| parent \| prev [-]
		> The nondeterminism and chaotic nature of LLM output is enough to rule them out as a reasonable foundation for any fully automated system that would be controlling anything potentially dangerous or damaging. You could say the same for humans. The difference is that humans have been conditioned to be extra cautious about things that could get them fired, and there is no benchmark for Meta's model developers to benchmax about that.