Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).

Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.

I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.

▲

xg15 2 hours ago | parent | next [-]

> avoid a giant prefix miss

I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.

Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.

	▲	reactordev an hour ago \| parent [-]
		This is the approach I took. Before leaving scope, results are filtered to remove tool call errors and bloat and instead append those errors to a different system. A much smaller tool call error result is returned to the agent while keeping the reasoning for failure intact.

▲

Muromec 4 hours ago | parent | prev | next [-]

>some cases, tools are added mid-run to avoid a giant prefix miss

Wait, can you actually do that? The model should not exactly care where in the context the tools are, but I haven't realised it's a thing that works

	▲	gpugreg 3 hours ago \| parent [-]
		Context is just one long string. You can do whatever you want with it. Training helps to conform to specific syntactic idiosyncrasies, but LLMs are large enough these days that they can follow pretty much any instructions most of the time. For example, if you want to use an LLM API that does not natively support tool calls, you can tell the LLM to emit the tool calls for your preferred tools in a markdown code block and parse them yourself.

▲

spankalee 4 hours ago | parent | prev [-]

What would a "fully-constrained tool call environment" be?