We are also interested in this and may focus on it in a future post. They are two different perspectives: (1) What is the overall quality of LLM+serving stack+harness, and (2) How close does the behavior of our setup match the first-party provider's? In this post, we focused on the latter. Moonshot AI, the creators of Kimi, recommend constrained decoding to be on by default, and also serve Kimi that way. This implies that it helps overall performance, though it does not provide numbers for it. We want to serve Kimi as recommended, and ran into the challenges described in the post.

There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3]. Overall, it seems to me that there is a trend towards using constrained decoding. OpenAI's newer Responses API has constrained decoding turned on by default as well, similar to Moonshot AI's Kimi.

1: https://arxiv.org/abs/2408.02442

2: https://blog.dottxt.ai/say-what-you-mean.html

3: https://dylancastillo.co/posts/say-what-you-mean-sometimes.h...