Why not have a second model that reads every response looking for something nefarious?
Sounds like a SIEM to me
Who watches the watcher?
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.