Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)

This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.

I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.

Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.

The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D

I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)

▲

serf 7 hours ago | parent | next [-]

high tps and braindead model is a dangerous and fun thing to watch.

give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.

▲

clhodapp 7 hours ago | parent | prev [-]

At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?

It feels like that would be the expected result, all else equal.

▲

Imustaskforhelp 7 hours ago | parent [-]

I think it's because of the fact that Deepseek at 16B can be comparable enough in tasks to Mimo with 42B

I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.

Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]

I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.

Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]

[0]: https://artificialanalysis.ai/models/comparisons/nemotron-3-...

[1]: https://artificialanalysis.ai/models/k2-horizon-375b-a23b & https://ifm.ai/blog/k2/

▲

clhodapp 6 hours ago | parent [-]

I guess what I'm really saying is just that being impressed with a model's size and being impressed with a model's speed are roughly the same thing for similar architectures and hardware.

	▲	Imustaskforhelp 5 hours ago \| parent [-]
		Yes you are correct about it, though aside from a minor differences within the architecture itself, its right that being impressed with token speed is being impressed with model's size. So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D