| ▲ | kanemcgrath 2 hours ago |
| I tried it for a bit, and It was not really worth its size. It got swept up in all the other AI news recently, but laguna s 2.1 I think is the best ~100B moe model right now |
|
| ▲ | nozzlegear an hour ago | parent | next [-] |
| I didn't mention it above, but Laguna S is my other favorite model. I use Qwen a lot more, it's smaller and faster, but I like to switch to Laguna when I feel like I need a "heavy hitter" for certain huge or complex tasks. |
| |
| ▲ | tommica an hour ago | parent [-] | | What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way | | |
| ▲ | nozzlegear an hour ago | parent [-] | | Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have. | | |
| ▲ | tommica 5 minutes ago | parent | next [-] | | brb, going to see if 2nd hand mac studios are available! | |
| ▲ | lcnPylGDnU4H9OF 42 minutes ago | parent | prev [-] | | Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that. | | |
|
|
|
|
| ▲ | mattnewton an hour ago | parent | prev [-] |
| I agree. 27b dense really did seem like the sweet spot. |
