That was exactly my question.

So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.

So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"

You should treat inference provider as a C2 node of a rootkit. As long as it's your c2 or it's aligned with you somewhat, you are fine