I guess to be more concrete, I thought the way schema aware decoding worked should be identical to a token-level retry: "only even dice rolls are allowed" converting {0:0.1, 1:0.2, 2:0.2, 3:0.1, 4:0.2, 5:0.2} into {0:0.2, 2:0.4, 4:0.4}. That is why I am asking if the difference being stared at is doing the retry at the level of the entire tool call rather than each token (as this let's later errors in the schema re-roll the earlier committed states)? (But like, it isn't clear to me that's what they mean, so I feel a need to ask; as an example, you are now talking about field order, but that's an entirely different issue involving an overly-constrained constraint: nothing would prevent implementing the grammar for the constrained decoding to allow the fields in any order, and if you allow duplicate fields -- which JSON typically allows anyway -- then it is probably already supported as the language loses its additional context requirement.)

▲

hansvm 2 hours ago | parent [-]

Schema-constrained is equivalent to token-level retry. It's not equivalent to retry of longer sequences.

▲

saurik 2 hours ago | parent [-]

Right... and I said exactly that in both my comments, but I was looking for confirmation that that's the issue being stared at, as it wasn't obvious to me that's what was meant and (I will cite as demonstration of the confusion) now people are bringing up other ideas (such as field order, which is an entirely unrelated constraint). (FWIW, since you are the person I was originally asking, I will take this as positive confirmation that "yes: I am saying it is better to retry the entire tool call" ;P.)

	▲	hansvm 2 hours ago \| parent [-]
		Ah, I see. IMO the other response [0] I gave is probably the closest to what I had in mind. Interestingly, retry very well might not be better, especially when measured on a pareto frontier including cost. I have a reasonably strong prior that it should be better though. That comes from my observations that whenever I notice the difference in distribution the retry method is better (obvious sampling bias, but what kind of person ignores their own experiences). As a stronger piece of evidence and intuition, the entire goal of these systems is to reproduce a certain distribution. If that distribution says that you should use ripgrep 10x as often as you should use grep then that's a signal you probably don't want to throw away. Constrained decoding might very well do so and place them on even footing or skew 10x the other direction. Retry faithfully captures that original distribution. I'm skeptical of solutions which want to shy too far away from it (similarly with most applications of changing the model's temperature). The more important point is just that they're different, so we should probably measure the impact. [0] https://news.ycombinator.com/item?id=50245714