| ▲ | nozzlegear an hour ago | |||||||
Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have. | ||||||||
| ▲ | tommica 5 minutes ago | parent | next [-] | |||||||
brb, going to see if 2nd hand mac studios are available! | ||||||||
| ▲ | lcnPylGDnU4H9OF 42 minutes ago | parent | prev [-] | |||||||
Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that. | ||||||||
| ||||||||