The fact is that humans are accountable and this, alongside training, makes it easy to align them to your own goals.

There’s always the possibility of rogue individuals (recent Apple incident), but the likelihood is very low. If you have a DBA that have write access to the prod DB, you don’t fear that a random text somewhere could trigger the deletion of your customers table. Because the DBA will self regulate (with the help of processes) to not do that.

▲

TeMPOraL 3 hours ago | parent [-]

Right. But even with a DBA, the possibility remains. We accept that.

That's kind of my point with fighting against the "lethal trifecta" and "code vs data" mindset - once people engage cybersecurity mindset, they're all binary, "a system is either perfectly safe or is broken". With general AI - LLM or whatever comes next - you'll never have "perfectly safe". So the focus should be to either drive the risk down to minimum - like we do with people - or just not use LLMs for a task in the first place.

Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.

	▲	skydhash 2 hours ago \| parent [-]
		> Right. But even with a DBA, the possibility remains. We accept that. You're forgetting the element of scale and replication. How easy it is to bribe a DBA of a major platfoms like Gmail? How easy to replicate the same destructive behavior to other DBA? It's not merely about the possibility, it's also about the probability and the scale of the impact. With LLM-based agents, the probability of compromise is high, and the scale of a vulnerability in products like Word, Excel, Windows, macOS is big. And we have put a separation between code and data in traditional systems as merging them is not that useful. > Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation. The issue is not the LLM. The issue is the harness those products wraps the LLM in and insist on making tools act according to the LLM's output. Having unreliable (as in uncontrollable) output be the control plane of tools is the issue here. Both the LLM input (prompt+user data) and the output should've stayed in the data plane and not move in the control plane.