| ▲ | XCSme 3 hours ago |
| The new Meta 30B models seems A LOT better: https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi... |
|
| ▲ | thehamkercat 3 hours ago | parent | next [-] |
| Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old) but Qwen 3.8 27B is dropping this week... |
| |
| ▲ | XCSme 2 hours ago | parent [-] | | Yes, I was surprised to see doing it as well as Qwen 3.7 27b. Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model. Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results. |
|
|
| ▲ | eli 3 hours ago | parent | prev | next [-] |
| The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all. |
| |
| ▲ | XCSme 2 hours ago | parent | next [-] | | I should add a F.a.q. for this question. The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes. Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice. But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended). | |
| ▲ | markasoftware an hour ago | parent | prev [-] | | yep, the person you're responding to created the benchmark and is using HN comments as advertisement. | | |
| ▲ | XCSme 31 minutes ago | parent [-] | | I did, avertisment is a big word, as I gain nothing from the traffic, I run the website for myself, and some other people find it useful too. Happy to hear what would make the website more useful. |
|
|
|
| ▲ | khimaros an hour ago | parent | prev | next [-] |
| lightning is sparse, glimmer is dense |
| |
| ▲ | XCSme 31 minutes ago | parent [-] | | Oh, good to know, I just quickly tested and published the results. I will add model sizes (total/active params) for each model, good point. |
|
|
| ▲ | Tactical45 3 hours ago | parent | prev | next [-] |
| At what cost difference? |
| |
| ▲ | XCSme 3 hours ago | parent [-] | | I don't think it matters, if it's for local/on-device usage. The cost is similar vram footprint I guess (?) | | |
| ▲ | sleepyeldrazi 19 minutes ago | parent [-] | | loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test). As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token. | | |
| ▲ | XCSme 2 minutes ago | parent [-] | | I am asking mostly for running on a 3090. I think the tps difference between them (both fitting in vram) won't be more than 2x in practice. I would happily take 20tps over 40tps, if the model gets 3x more correct answers. |
|
|
|
|
| ▲ | rllearneratwork an hour ago | parent | prev [-] |
| and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW |
| |
| ▲ | XCSme 30 minutes ago | parent [-] | | Is that the case? If the entire model fits in vram, won't the tps be comparable? |
|
