Remix.run Logo
sznio 5 hours ago

I don't work at Anthropic, but I would assume they could serve smaller quantizations during peak hours - this effectively controls the "resolution" of the model. They could also control the resolution of the KV cache, which would make the model not necessarily dumber, but worse at understanding the incoming requests. And finally, you could pass off what was "high" effort as "extra", because why not.