| ▲ | nextaccountic 4 hours ago | ||||||||||||||||
> The model invents new categories (e.g. apartments) and doesn’t stick to the provided list of allowed categories Can this specific failure mode be solved by providing a grammar that the output must adhere to? (Not sure if Qwen has this feature, it's used for eg. to ensure the output is parseable json) | |||||||||||||||||
| ▲ | thomascountz 2 hours ago | parent | next [-] | ||||||||||||||||
Yes, you can use constrained decoding like logit masking to force all invalid tokens in the vocabulary to -inf, and effectively be removed from selection. I believe llama.cpp exposes this by accepting a formatted grammar. | |||||||||||||||||
| ▲ | nl 4 hours ago | parent | prev [-] | ||||||||||||||||
It can. It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself. Note that it is hard to make work if you turn thinking on because the grammar gets complicated quickly (I don't recall if Qwen 0.6B can do thinking). | |||||||||||||||||
| |||||||||||||||||