> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read

I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.

Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod.

As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.

In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window.

My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.