Remix.run Logo
▲ febed 3 hours ago

Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?

▲jesdo 3 hours ago | parent [-]

Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.