| ▲ | walrus01 10 hours ago | |
It might be less about quality degrading, but on multi-user platforms running the model, they have an economic incentive to have each user not fill up the full size of the context cache. Filled context cache being held in GPU RAM is context cache RAM that isn't available to other users. If the model is instructed to periodically ask the user to start from a clean slate context, and some users do comply with that, they probably have good stats on average size of context cache use for users who are presented with that answer (vs users who are not), basic A/B testing stuff. Might also be performance related in tok/s for what users will perceive as a more speedy experience. For a much smaller scale example, compare local performance of qwen 3.6 27B (not MoE) Q8 with 250,000+ context available, run on local hardware, tok/s generation rate when context is empty vs when context used is at 95,000. Same principle will apply to a much larger model. | ||