(Disclosure: I work for Microsoft) I run automated red-teaming on my RAG samples through the azure-ai-evaluation SDK, which uses an adversarial LLM (an LLM without the guardrails) plus the pyrit package to come up with horrible questions to ask your app and then transform them (base64, ceaser cipher, urlencode, etc), to see how the app will respond. It's really interesting to see the results, and I agree that red-teaming generally can be a good use of LLMs.

Video of me demo'ing it here: https://www.youtube.com/watch?v=sZzcSX7BFVA (Sorry I'm shout-y, weird venue)

Any experiences like this one? https://www.lesswrong.com/posts/MnYnCFgT3hF6LJPwn/why-white-...