| ▲ | anana_ 5 hours ago |
| For more context, this puts it on par with models like GLM 5.2 and GPT 5.6 Luna, which are far larger |
|
| ▲ | anana_ 4 hours ago | parent | next [-] |
| And to read the tea leaves a little: 3.8 actually performs slightly worse than 3.6 on AA-Omniscience Accuracy, which could imply that they traded out world knowledge for capability in other areas. It also produces nearly twice as many tokens per task as 3.6 (and by extension, time), which may be a tradeoff required to achieve correctness at this parameter size. |
| |
| ▲ | skohan 4 hours ago | parent [-] | | Imo it makes sense for things to move in the direction of small, focused models that excel in one area. I use LLMs for technical work 99% of the time, I could care less about general world knowledge, or if the model is good at creative writing. With good orchestration and delegation you can get surprisingly far with small models running on consumer hardware. | | |
| ▲ | tancop 3 hours ago | parent | next [-] | | The biggest untapped market is pure agentic models that are built for tool calling and non hallucination instead of memorizing facts. You need some world knowledge (as in common sense) to build a useful model, but I don't think perfect recall on general QA is a good use of space when you have web search and structured knowledge in Wikidata or Wolfram Alpha. Training should focus on tasks that require real intelligence instead of memory. Creative writing is actually good for this if you score it on coherence instead of getting random real life details right. Basic level of coding (simple prompt to code, don't need to one shot complex projects) is also great because writing a small script is more efficient than 20 separate tool calls. | | |
| ▲ | CamperBob2 15 minutes ago | parent [-] | | With respect to Wolfram Alpha, it's worth noting that VibeThinker-3B is basically a match for the larger frontier models -- hundreds of times larger -- in the narrow domain of mathematical and logical reasoning problems. It doesn't seem necessary to resort to external models or tools for that, at least in principle. |
| |
| ▲ | anana_ 4 hours ago | parent | prev | next [-] | | Agreed. Luckily, this model also scores high in AA non-hallucination, so it knows what it doesn't know -- perfect for situations where it can just tool call a web search. | |
| ▲ | drob518 3 hours ago | parent | prev [-] | | Yep, exactly. I keep saying that I want the “coding expert” extracted from these multi-T parameter models to run locally on reasonable hardware (large laptops, not servers). Yea, I know there’s no single “coding expert” that you can actually extract in these models, but you get what I mean. Like you said, when I’m coding, I don’t care about world knowledge, and I’m fine with consulting another model when I need that. |
|
|
|
| ▲ | bertili 4 hours ago | parent | prev | next [-] |
| And more context: Same score as the latest DeepSeek Flash 0731 which has 284B parameters! (13B active) Its also the second best Qwen model, much better than Qwen 3.7 Max, but significantly below Qwen 3.8 Max. |
| |
| ▲ | anthonypasq 4 hours ago | parent [-] | | isnt the active parameter count more relevant than the total? qwen is a dense model no? | | |
| ▲ | kzrdude 3 hours ago | parent [-] | | For some tasks yes, and we don't know how many active parameters Luna is using..probably less than 27B |
|
|
|
| ▲ | nsingh2 4 hours ago | parent | prev | next [-] |
| Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment. |
| |
| ▲ | sottol 4 hours ago | parent | next [-] | | I'm torn on this - on the one hand performance matters, on the other so does capability. I could run Qwen 3.6 27B on my laptop, but at 5 tok/s it was too slow even without overthinking - I never used it. OTOH, Qwen 3.6 35B A3B ran at 20 tok/s but it just could not get done what I asked of it. It sort of got close but you had to repeat and retry so much that it might have been faster to run 27B dense... maybe? So that said, I might take a much better model that runs 2-3x slower (total time per task) but that's more capable over a faster, less capable one. I'd also like to try a proper "plan-then-execute" type execution where thinking is entirely disabled (or low) during the execution stage but enabled/max during the planning stage. I will definitely give 3.8 27B a better shot than 3.6 though. | |
| ▲ | skohan 3 hours ago | parent | prev [-] | | Depends on your use-case. Over the past couple days, I've found 192k context more than enough for coding. There's more thinking for sure compared to comparably sized models (running on xhigh), but I've found the results are so much better that the entire session consumes less tokens on average since weaker models need more review passes. |
|
|
| ▲ | 4 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | johnnyApplePRNG 4 hours ago | parent | prev | next [-] |
| We don't actually know how large they are, actually. |
| |
| ▲ | halJordan 4 hours ago | parent [-] | | Well, ackshually. E do know exactly how big glm 5.2 is. And there's more than enough data to draw conclusions about luna. Or are you one of the guys who says "big bang is just a theory"? | | |
|
|
| ▲ | catigula 3 hours ago | parent | prev [-] |
| Which should tell you how useful these benchmarks are. |
