Is the schema constraint not implemented in a way where the new distribution is identically scaled to the old distribution with retry? I had thought the way it worked was just to disallow invalid next states... oh, or is the idea that you think it is much preferred to retry at the level of the entire tool call, rather than (conceptually) per token? I guess I can see that making a lot of sense, as maybe the model gets "backed into a corner", so to speak, with a really bad initial choice and is then trying to rescue horrible states?

▲

hansvm 2 hours ago | parent | next [-]

The way it works is to disallow invalid next states, and getting backed into a corner is one way to think about one possible failure mode.

For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like

{"name": "Bob", "biography": "Bob is a fantastic builder, having ...

Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.

As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.

For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.

▲

Creamsicle47 3 hours ago | parent | prev [-]

What the other guy means is the model is forced to decode potentially out of distribution and subsequent parts of the sequence will be sampled from a higher-entropy distribution.

i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.

You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.

▲

saurik 2 hours ago | parent | next [-]

I guess to be more concrete, I thought the way schema aware decoding worked should be identical to a token-level retry: "only even dice rolls are allowed" converting {0:0.1, 1:0.2, 2:0.2, 3:0.1, 4:0.2, 5:0.2} into {0:0.2, 2:0.4, 4:0.4}. That is why I am asking if the difference being stared at is doing the retry at the level of the entire tool call rather than each token (as this let's later errors in the schema re-roll the earlier committed states)? (But like, it isn't clear to me that's what they mean, so I feel a need to ask; as an example, you are now talking about field order, but that's an entirely different issue involving an overly-constrained constraint: nothing would prevent implementing the grammar for the constrained decoding to allow the fields in any order, and if you allow duplicate fields -- which JSON typically allows anyway -- then it is probably already supported as the language loses its additional context requirement.)

▲

hansvm 2 hours ago | parent [-]

Schema-constrained is equivalent to token-level retry. It's not equivalent to retry of longer sequences.

▲

saurik 2 hours ago | parent [-]

Right... and I said exactly that in both my comments, but I was looking for confirmation that that's the issue being stared at, as it wasn't obvious to me that's what was meant and (I will cite as demonstration of the confusion) now people are bringing up other ideas (such as field order, which is an entirely unrelated constraint). (FWIW, since you are the person I was originally asking, I will take this as positive confirmation that "yes: I am saying it is better to retry the entire tool call" ;P.)

	▲	hansvm 2 hours ago \| parent [-]
		Ah, I see. IMO the other response [0] I gave is probably the closest to what I had in mind. Interestingly, retry very well might not be better, especially when measured on a pareto frontier including cost. I have a reasonably strong prior that it should be better though. That comes from my observations that whenever I notice the difference in distribution the retry method is better (obvious sampling bias, but what kind of person ignores their own experiences). As a stronger piece of evidence and intuition, the entire goal of these systems is to reproduce a certain distribution. If that distribution says that you should use ripgrep 10x as often as you should use grep then that's a signal you probably don't want to throw away. Constrained decoding might very well do so and place them on even footing or skew 10x the other direction. Retry faithfully captures that original distribution. I'm skeptical of solutions which want to shy too far away from it (similarly with most applications of changing the model's temperature). The more important point is just that they're different, so we should probably measure the impact. [0] https://news.ycombinator.com/item?id=50245714

▲

Aurornis 2 hours ago | parent | prev | next [-]

> or perhaps adjusting the order of the json output affects it to produce lower-quality results.

The order of the output fields can actually be used to tune performance of the outputs. It can be helpful to force the LLM to reason about specific fields first if later fields benefit from seeing those earlier fields' output. You have to test it because some times the results are surprising in directions you didn't expect.

XGrammar has an option somewhere to allow fields to appear in any order but I don't know if vLLM exposes it. There are some tradeoffs and limitations for allowing fields to show up in any order because there is some complexity in schema parsing like that.

▲

hashmap 2 hours ago | parent | prev [-]

for putting together something like a tool-call decision model, the conditioning will take less time than getting the data for it, which also will not take much time if you keep your session data. whatever fails is your negative contrast, whatever passes is your positive data. the space of tool calls your agents will use in your particular environment is not actually that big, which makes this one of the easiest wins you can pull off.