Btw - assuming NeurIPS reviews aren’t garbage tomorrow, I’ll have a paper out which claims that most long context problems in models are really sampling problems in disguise

Switch to a modern sampler like min_p or ideally a better one like top-n-sigma (it’s in llamacpp) and your “my model gets stupid at long context problems” will basically go away.

Unfortunately this fact is still not well appreciated yet despite nearly every modern sampling technique getting an oral wherever they get presented. Min-K just got an oral at ACL 2026, for a hyper recent example of this. There’s a reason they keep getting orals.

The field massively ignored sampling for mostly safety reasons and now the whole field incorrectly believes long context doesn’t work on small models. Long context is an out-of-distribution problem. Your sampler configured properly keeps you in distribution.

Oh and this is doubly true for quantized models. I run my qwen 3.6 27b with 4bit quants from unsloth and get excellent performance because my sampler stack is good and not the garbage that is top_p and top_k. Also, yes, you need to ignore the trash recommended sampler settings from the Chinese labs (they’re wrong/bad).

Interesting. I am not familiar with model internals at this level because I have only been working at the application layer so far, but will definitely research this further. When you get the paper published would appreciate if you can drop a comment with the link so I can read it.