One thing that especially interests me about these prompt-injection based attacks is their reproducibility. With some specific version of some firmware it is possible to give reproducible steps to identify the vulnerability, and by extension to demonstrate that it's actually fixed when those same steps fail to reproduce. But with these statistical models, a system card that injects 32 random bits at the beginning is enough to ruin any guarantee of reproducibility. Self-hosted models sure you can hash the weights or something, but with Gemini (/etc) Google (/et al) has a vested interest in preventing security researchers from reproducing their findings.

Also rereading the article, I cannot put down the irony that it seems to use a very similar style sheet to Google Cloud Platform's documentation.