What are you running that model on?

A 3 bit quant will run on a 128gb MacBook Pro, it works pretty well.

	▲	nl 32 minutes ago \| parent [-]
		A 3 bit quant is quite a lot weaker than the OpenRouter version the OP is using.

I just use openrouter, it's free for now. But I would pay 30-100$ to use it 24/7.

	▲	aappleby 2 hours ago \| parent [-]
		Ah, I thought you meant you were running it locally.