The point I was getting at was more of a question of "Okay, how can this be used adversarially?" or "How can we break this?".

"What can a malicious inference engine do"

Something like that

▲

NitpickLawyer 5 hours ago | parent [-]

Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?

A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...

▲

hypfer 5 hours ago | parent [-]

That was exactly my question.

So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.

So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"

	▲	Muromec 5 hours ago \| parent [-]
		You should treat inference provider as a C2 node of a rootkit. As long as it's your c2 or it's aligned with you somewhat, you are fine