I have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).

▲

monster_truck 2 hours ago | parent | next [-]

One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.

4 bit quants can be fine, try it and see.

▲

segmondy 2 hours ago | parent | prev | next [-]

4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...

▲

himata4113 3 hours ago | parent | prev | next [-]

use MoE offloading

▲

echelon 3 hours ago | parent | prev [-]

> we have restrictions that we can't use a chinese model!

Please expand on this if you can. Why can't you? Who mandated this?

▲

derektank 2 hours ago | parent | next [-]

Last I heard, US defense contractors had been directed to not use Deepseek in their operations late last year[0], and most have taken that as an indicator that they shouldn’t be using chinese models generally.

[0] https://digitalpolicyalert.org/change/20659-physical-and-cyb...

▲

a012 2 hours ago | parent | prev [-]

US politics. Lot of US corps now ban Chinese models, don’t ask me why, the people in charge are just dumb fucks

	▲	alembic_fumes 2 hours ago \| parent \| next [-]
		Seems like a reasonable policy to me, depending on the sector at the least? Even with open weight / self-hosted models, there's no way to know if the model has been fine-tuned to contain malicious behaviour, triggered only within very specific conditions. In fairness this doesn't warrant the way that the policy targets Chinese models specifically, but Chinese providers do fall under the broader category of entities that cannot be prosecuted by the U.S. law if such behaviour is found.
	▲	mapontosevenths 2 hours ago \| parent \| prev [-]
		Executives live in a weird bubble driven largely by consulting firms. Right now those firms are spreading FUD about Chinese models.