This is a thing! For example, https://arxiv.org/abs/2511.06516
that's brilliant, I wonder why we haven't seen much use of it to do very heavy quantization