They’re essentially being used as a programming language interpreter. This attack could easily have been done with Python or Ruby or Perl. There can’t be a realistic expectation that these tools are robust against malicious input. You have to either sandbox them or keep malicious input away from them.

> Should the AI Assistant NOT reply to the request it was given? Why shouldn't it?

LLMs are not a dumb interpreter. At minimum, they are a client-server architecture that can be used as a control plane. But they are much more than that and can likely employ advanced detection and classification heuristics.

The vendors have the capability of (1) stopping this in its tracks, (2) understanding the extent of the attack and notifying customers, (3) studying the breadth of approaches used (4) for future, more ambitious attacks, monitoring them live as threat actors explore systems.

Google and Anthropic absolutely have responsibility here and must devote resources to this.

I am shocked that this is being met with such hostility. I cannot picture a world where LLM vendors are not responsible for making a best attempt at safeguarding their customers. Especially as they seek to have a greater role in business and financial automation.

I've worked at fintechs and we had to go out of our way to look out for our customers. We purchased compromised password and email lists and scanned for impacted customers. Our business didn't cause the data breaches, but we viewed it as our responsibility to protect customers.

Google and Anthropic have the greatest opportunity to make a difference here.

* THIS IS ABSOLUTELY A SEV0 FOR GOOGLE AND ANTHROPIC *:

While it's not a systems outage, it has incredible potential to shape future business and market sentiment. There are going to be major articles written about this in every publication you can think of. Publications that business decision makers read. Forbes, the New York Times, Business Insider. And Google and Anthropic are going to want to own their blurb and state that they acted fast and responsibly. If they're lucky, they can even spin this as an opportunity.

This is the difference between LLMs being allowed in the business workplace and being met with increasing scrutiny. (Not that they shouldn't be scrutinized, but that this incident will overwhelmingly shape the future of the decision envelope.)

My first two comments in this thread were my initial reaction to what was happening.

I made the above, longer form post to hopefully grab the attention of Google and Anthropic folks. My top-level posts always fall to the very bottom of the page.

Google and Anthropic need to be tracking this.