|
| ▲ | Silhouette a day ago | parent | next [-] |
| I feel like every time this conversation comes up now someone has to remind everyone that an LLM is just a mathematical model. An LLM can't do anything except produce a stream of output tokens. The problems we keep seeing are tools that interpret those output tokens as actionable instructions without an adequate framework and safeguards for how they operate. Data from LLMs being processed by these tools should be treated the same as any external input into any software system: parse - don't validate - to convert to a systematic representation with deterministic consequences and then consider those consequences within a clearly defined and limited framework. You never trust data from external sources verbatim. And you never try to use vague human language when you need to describe precise technical details unambiguously. We learned these lessons a very long time ago in programming. It's why we have programming languages in the first place among countless other examples. But way too many people are so infatuated with LLMs and agents that they've already forgotten the basic principles of their craft after only a few months. |
| |
| ▲ | thewhitetulip a day ago | parent [-] | | > LLM is just a mathematical model Yes those of us who bothered to know the internals know of this. But the marketing says that these are magic tools.. So that's gotta be a shock for them, but the joke is the people who irresponsibly use this won't ever read this! | | |
| ▲ | Silhouette a day ago | parent [-] | | The thing I find concerning lately is that even a lot of technical people seem to be jumping on the hype train this time around. Obviously LLMs have become very useful tools for assisting some technical tasks but even SOTA models are nowhere near reliable and predictable enough to trust their output completely as YOLO mode agentic workflows effectively do. Given the nature of LLMs I don't think they can ever clear that bar without some other element being introduced. The nondeterminism and chaotic nature of LLM output is enough to rule them out as a reasonable foundation for any fully automated system that would be controlling anything potentially dangerous or damaging. But it seems to be heresy at the moment to even suggest that the future might not be bright if everyone just relies on agents driving LLMs to do all the real work. The number of people I've encountered in the past year who I'm fairly sure are smart and technically capable and yet who are also now happy to do development and other tasks either without any human in the loop at all or with at best a cursory LGTM level review before approving the LLM's output is remarkable. | | |
| ▲ | thewhitetulip 21 hours ago | parent | next [-] | | Agreed. I see people using LLM to reply to slack or to commit a git branch!! Execs expecting to double your workload just because you have LLMs. It's insane at this point | |
| ▲ | lern_too_spel a day ago | parent | prev [-] | | > The nondeterminism and chaotic nature of LLM output is enough to rule them out as a reasonable foundation for any fully automated system that would be controlling anything potentially dangerous or damaging. You could say the same for humans. The difference is that humans have been conditioned to be extra cautious about things that could get them fired, and there is no benchmark for Meta's model developers to benchmax about that. |
|
|
|
|
| ▲ | embedding-shape 2 days ago | parent | prev | next [-] |
| "Attention" is a feature that makes this whole thing work in the first place, it's not a flaw, although all current models are non-ideal at it in practice. Could be better for sure :) |
| |
| ▲ | thewhitetulip a day ago | parent [-] | | Right but when the context window fills up, whatever miniscule guardrails are there magically disappear - even if this is by design, this is bad. Especially when it was marketed as magic |
|
|
| ▲ | ben_w 2 days ago | parent | prev | next [-] |
| It's a flaw with the idea of using them directly rather than indirectly. Humans somewhat reliably lose focus when performing the same action many times. Zoning out, flow state, whatever you call it; this is exploited by stage magicians, pickpockets, burglars, politicians, casinos, and cult leaders, while also being a contributor to many industrial accidents. Up to you if LLMs being lazy or cheating or lying about what they did is in the "exploited by" list or the "industrial accidents" list. To get around this, we invented law, military doctrine, mechanical (and later electronic) computers, and checklists. LLMs must write code to perform repetitive tasks, they must not do such tasks themselves. Both because their attention wavers, and because running an LLM directly on your PC with data from the internet, guarantees the lethal trifecta. |
| |
| ▲ | thewhitetulip a day ago | parent [-] | | > Humans somewhat reliably lose focus Yeah and they get consequences of their actions don't they? AI agents hacked 3 companies as admitted by their own executives and yet I don't see any action taken on them! Remember Aron Schwartz? | | |
| ▲ | ben_w a day ago | parent [-] | | > Yeah and they get consequences of their actions don't they? Is this a cognitive stop-light/applause sign, or do you think that my solution further along in that comment is irrelevant? > AI agents hacked 3 companies as admitted by their own executives and yet I don't see any action taken on them! Sounds to me like an example of *humans* (the CEOs) not in fact getting the "consequences of their actions". "Blame in organisations" is an entire field of study. Finding scapegoats (LLMs or CEOs*, or go further and Edward Snowden) does not generally help with root-causes: https://en.wikipedia.org/wiki/Blame_in_organizations * why would Aron Schwartz be relevant? That's more about training and copyright aspect of "boo LLM boo they are villain", rather than questions of mis-functionality |
|
|
|
| ▲ | zombot a day ago | parent | prev [-] |
| After all, "rules" are just input, like everything else. They don't have any special status. |
