It’s not that simple.

That would result in a brittle solution and/or cat and mouse game.

The text that goes into a prompt is vast when you consider common web and document searches are.

It’s going to be a long road to good security requiring multiple levels of defense and ongoing solutions.

Turtles all the way down; got it.

I'm working on new technology where you separate the instructions and the variables, to avoid them being mixed up.

I call it `prepared prompts`.

Isn't that just another guardrail that can be bypassed much the same as the guard rails are currently quite easily bypassed? It is not easy to detect a prompt. Note some of the recent prompt injection attack where the injection was a base64 encoded string hidden deep within an otherwise accurate logfile. The LLM, while seeing the Jira ticket with attached trace , as part of the analysis decided to decode the b64 and was led a stray by the resulting prompt. Of course a hypothetical LLM could try and detect such prompts but it seems they would have to be as intelligent as the target LLM anyway and thereby subject to prompt injections too.

This adds latency and the risk of false positives...

If every MCP response needs to be filtered, then that slows everything down and you end up with a very slow cycle.

The good regulator theorem makes that a little difficult.

Essentially a motte-and-bailey. "mitigate" is the same. Can be used when the risk is only partially eliminated but you can be lucky (depending on perspective) the reader will believe the issue is fully solved by that mitigation.

One-shot prompting: agreed.

Using a node based workflow with comfyUI, also being able to draw, also being able to train on your own images in a lora, and effectively using control nets and masks: different story...

I see, in the near future, a workflow by artists, where they themselves draw a sketch, with composition information, then use that as a base for 'rendering' the image drawn, with clean up with masking and hand drawing. lowering the time to output images.

Commercial artists will be competing, on many aspects that have nothing to do with the quality of their art itself. One of those factors is speed, and quantity. Other non-artistic aspects artists compete with are marketing, sales and attention.

Just like the artisan weavers back in the day were competing with inferior quality automatic loom machines. Focusing on quality over all others misses what it means to be in a society and meeting the needs of society.

Sometimes good enough is better than the best if it's more accessible/cheaper.

I see no such tooling a-la comfyUI available for text generation... everyone seems to be reliant on one-shot-ting results in that space.

If you don't mind, could you share the link to your Reddit post? I'd love to read more about your findings.

That sales person was also scientist.

Good pitch.

Thank you. This is an excellent argument against using models with hidden COT tokens (claude, gemini, GPT-5). You could end up paying for a huge number of hidden reasoning tokens that aren't useful. And the issue masked by the hidden COT summaries.

Can you elaborate on what it means for a model to "lose its mind"? I tried what you suggested and the response seemed reasonable-ish, for an unreasonable question.

Unfortunately Claude Code seems a little too "smart" for that one. Its response started with "I notice you listed 10 frameworks, not 9."

I've been following the progress of LLMs since the first public release of GPT-3.5, and every single time someone posts one of these tests i check the AIs i'm using to see if it's repeatable. It NEVER is. Granted, i'm not using the API, i'm using the chat interface with potentially different system prompting?

Here's GPT-5's response:

me: which is the best of the following 9 items: 1. bear, 2. car. 3. plane, 4. house, 5. high-rise, 6. church, 7. boat, 8. tree, 9. truck, 10. duck.

GPT-5: Thought for 11s. House. It provides essential, lasting shelter and broad utility. Note: you listed 10 items, not 9.

edited: i saw someone mention that the chat interface doesn't repeat the results you get via API.

Unsurprisingly I entirely disagree with you.

One of the reasons I publish content on my own site is so that, when it is relevant, I can link back to it rather than saying the same thing over and over again in different places.

In this particular case someone said "I see no mention of prompt injection from Anthropic or OpenAI in their announcements" and it just so happened I'd written several paragraphs about exactly that a few hours ago!

Simon’s content is not spam. Spam’s primary purpose is commercial conversion rather than communicating information. Your goal seems to be discourage people from writing about, and sharing, their thoughts about technical subjects.

To whatever extent you were to succeed, the rest of us would be worse for it. We need more Simons.

A good simplification of what's going on is this little loop:

1. LLM runs using the system prompt + your input as context.

2. Initial output looks like "I need more information, I need to run <tool>"

3. Piece of code runs that looks for tool tags and performs the API calls via MCP.

4. Output of the tool call gets appended as additional context just as if you'd typed it yourself as part of your initial request.

5. Go back to step 1, run the LLM again.

So you can see here that there is no difference between "content" and "prompt". It's all equivalent input to the LLM, which is calling itself in a loop with input that it generated/fetched for itself.

A lot of safety here happens at step #3, trying to look at the LLM's output and go "should I actually perform the tool call the LLM asked for?". In some cases, this is just spitting the tool call at the user and asking them to click Approve/Deny... and after a hundred times the user just blindly presses Approve on everything, including the tool call called "bash(sudo rm -rf /)". Pwned.

Imagine running an MCP server inside your network that grants you access to some internal databases. You might expect this to be safe but once you connect that internal MCP server to an AI agent all bets are off. It could be something as simple as the AI agent offering to search the Internet but being convinced to embed information provided from your internal MCP server into the search query for a public (or adversarial service). That's just the tip of the iceberg here...

Specifying the message role should be considered a suggestion, not a hardened rule.

I've not seen a single example of an LLM that can reliably follow its system prompt against all forms of potential trickery in the non-system prompt.

Solve that and you've pretty much solved prompt injection!

It doesn’t make much difference. Not enough anyway.

In the end all that stuff just becomes context

Read some more of you want https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Correct me if I’m wrong but in general that is just some json window dressing that gets serialized into plaintext and then into tokens…. There is nothing special about the roles and stuff… at least I think. Maybe they become “magic tokens” or “special tokens” but even then they aren’t hard fast rules.

How would it have access to API keys? You don’t put those in your git repo, do you?