Remix.run Logo
▲ drdaeman an hour ago

Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.

▲smallerize 36 minutes ago | parent [-]

It is already quantized. You need to convert to GGUF format though. Did your assistant explain this wrong?

▲drdaeman 32 minutes ago | parent [-]

My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.