So if I understand it correctly, the tool calls are not actually done by the model?

In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?

Could something like that also be used in other contexts?

I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?

▲

NitpickLawyer 4 hours ago | parent | next [-]

> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?

Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).

So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).

The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).

> Could something like that also be used in other contexts?

Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").

▲

hypfer 4 hours ago | parent [-]

The point I was getting at was more of a question of "Okay, how can this be used adversarially?" or "How can we break this?".

"What can a malicious inference engine do"

Something like that

▲

NitpickLawyer 4 hours ago | parent [-]

Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?

A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...

▲

hypfer 4 hours ago | parent [-]

That was exactly my question.

So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.

So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"

	▲	Muromec 3 hours ago \| parent [-]
		You should treat inference provider as a C2 node of a rootkit. As long as it's your c2 or it's aligned with you somewhat, you are fine

▲

FearNotDaniel 4 hours ago | parent | prev | next [-]

Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.

[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...

▲

hypfer 4 hours ago | parent [-]

I think I might've phrased something in ways that were confusing to people, because now two people have explained tool calls to me for some reason?

	▲	FearNotDaniel an hour ago \| parent [-]
		Just trying to be helpful. I couldn’t see any other way to interpret the question below except as someone who doesn’t know how tool calls get done and who might therefore value a little insight. Didn’t mean to be patronising, apologies if I misinterpreted your question. > So if I understand it correctly, the tool calls are not actually done by the model?

▲

jesdo 5 hours ago | parent | prev [-]

Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.

	▲	hypfer 4 hours ago \| parent [-]
		Yeah I know what a tool call is.