Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?

▲

foo12bar 3 hours ago | parent [-]

Maybe ask a separate model to look for indications against every model response you receive whether it is nefarious would work better.

	▲	sweezyjeezy 2 hours ago \| parent [-]
		My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."