These nerf trackers do not work. I wish people would stop posting them.

They don't detect a quantized model until the quantization gets down to 4 bits or less.

Humans absolutely do detect quantization well before 4 bits...

Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.

But quantized local models also dip in evals.