The big worry about this is with increasingly hard to make but useful quantizations, such as nvfp4. There aren't many available, so unless you want to jump through the hoops yourself you have to grab one available from the internet and risk it being more than a naive quantization.