[delayed]

Some physical constraints do not allow for the best security, and those physical constraints will always win out in the real world. When presented with the pick two of three options of fast, cheap, secure/done right, fast and cheap will always win out.

The “new” phone system (SS7) still relies on implicit trust and a lack of security.

Meta adding an AI chat window in whatsapp and Microsoft adding copilot in every word document was not done with developers in mind, which is why they're missing a lot of power user features that they'd surely have if they were targeted at developers.

You're mistaking the majority of what you see (like Claude et al) with the majority of stuff that is out there. The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.

There's a (terrifyingly) large number of developers who don't qualify as "security-oriented programmers".

Code/data separation is at the heart of computer security in the same way slapstick comedy is at the heart of humor.

There's an endless supply of people who think they know what is Code and what is Data, and they're always arguing with others who also think that, and neither realize that Code/Data classification is an opinion, a perspective. It doesn't hold in general.

Having a separation like this makes sense for super narrow systems, where you can define the allowed and disallowed use cases, enforce the distinction (because it's not real - therefore you have to enforce it mechanistically within your system), and willing to accept that some useful operations will be denied by your system.

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems.

Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator.

So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message?

Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue?

Wisdom says to do the latter.

> A pure Harvard architecture machine has exactly that separation.

It emulates and enforces that separation. A mathematical abstraction of a Harvard architecture machine has that separation, the real machine merely emulates it, and is only able to do so within some specific constraints (such as: no one hooks up dedicated programmer to the chip, or no one undervolts or overheats the cheap in clever way, or no one takes a swing at it with an x-ray source, or...).

That's the other thing people forget here: we're emulating abstract mathematical universes with real atoms, and then we're stacking those abstractions within abstractions. There is a whole segment of computer security that deals with that. When we say "once attacker has physical access, it's game over", or even discuss "side channels", is when we briefly remember that computer systems live in physical world, and the rules of our carefully designed abstract universes don't hold when you're on the outside of them and reaching in.

> Which is almost never the case.

Well, the topic is about AI..

LLMs are.

> If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate

You think. But there are ways around that. How about a credible extortion message targeting specifically you, that is embedded somewhere on the data sheet? Suddenly, the data has become the instructions...

The code sheet says take some bits from the data sheet and interpret it as if it were on the code sheet.

> the interpreter that acts on those is a piece of classical code, and sits outside of the model.

Correct, but it's an LLM that's reasoning about what stream of interpretable tokens should be emitted. The interpreter can certainly apply some security measures around what's being asked of it (like ask for confirmation), but that can only go so far. Is the human in the loop always capable of understanding what's safe to execute? If not, should we pass it through another fallible LLM to help make that judgement call?

Some security measures can be handled in a purely deterministic manner. But not all of them, and that's the problem.

I almost agree. Have you tried to add an image in LaTeX that does not wander to a random page?

  \begin{figure}[HERE!!!!!!]

And in the old compiler, I remember a problem with bounding boxes, and keeping a eps and pdf version of each image to get a correct dvi and pdf. I think this part is fixed now.

He probably means JSX mixing HTML with Javascript...

function Greeting({ name }) { return <h1>Hello, {name}</h1>; }