| ▲ | pettijohn 2 hours ago |
| 35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter, so I daily drive that. I wish there was a ~100B MoE with maybe 10B active. It would be super smart and fast! |
|
| ▲ | mattnewton 2 hours ago | parent | next [-] |
| There was a 3.5 122B 10A release - https://huggingface.co/Qwen/Qwen3.5-122B-A10B |
| |
| ▲ | kanemcgrath 2 hours ago | parent [-] | | I tried it for a bit, and It was not really worth its size. It got swept up in all the other AI news recently, but laguna s 2.1 I think is the best ~100B moe model right now | | |
| ▲ | nozzlegear an hour ago | parent | next [-] | | I didn't mention it above, but Laguna S is my other favorite model. I use Qwen a lot more, it's smaller and faster, but I like to switch to Laguna when I feel like I need a "heavy hitter" for certain huge or complex tasks. | | |
| ▲ | tommica an hour ago | parent [-] | | What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way | | |
| ▲ | nozzlegear an hour ago | parent [-] | | Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have. | | |
| ▲ | tommica 4 minutes ago | parent | next [-] | | brb, going to see if 2nd hand mac studios are available! | |
| ▲ | lcnPylGDnU4H9OF 41 minutes ago | parent | prev [-] | | Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that. | | |
|
|
| |
| ▲ | mattnewton an hour ago | parent | prev [-] | | I agree. 27b dense really did seem like the sweet spot. |
|
|
|
| ▲ | nozzlegear 2 hours ago | parent | prev [-] |
| I've heard 27B is smarter! I tried it some time ago but couldn't get it working with my oMLX. I need to try it again. |
| |
| ▲ | mattnewton an hour ago | parent | next [-] | | Honestly the 27b dense one punches way above its weight in a lot of domains, especially coding in my testing, so I think you will probably be disappointed. | |
| ▲ | npodbielski 26 minutes ago | parent | prev [-] | | In my case I would say they are comparable but moe models are looping and getting lost a lot more than dense models. On the other hand having 90t/s with any local model is nice and Pi with loop police extension can prevent looping a lot. |
|
