That's literally what the Anthropic paper shows. This isn't theoretical it's literally just what often happens irl if you put an LLM in this situation.