What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way

▲

nozzlegear an hour ago | parent [-]

Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have.

▲

tommica 6 minutes ago | parent | next [-]

brb, going to see if 2nd hand mac studios are available!

▲

lcnPylGDnU4H9OF 42 minutes ago | parent | prev [-]

Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.

	▲	tommica 6 minutes ago \| parent [-]
		Oh, that is a useful rule to know! Thanks!