But I’m not missing the point. If you can run one frontier model at 750t/s, then you can probably run many many instances of an SLM in parallel at a rate that exceeds 15k/s. That’s kinda the point of the flash or ultrafast variants. And they’re on something much more modern than llama3.1.

▲

windexh8er 2 hours ago | parent [-]

Yes, you are missing the point. 1) It's a demo. [0] 2) It hasn't been updated for 4+ months.

You don't need LLMs for everything. That is 100% the point. You can burn down the world with all of your frontier LLMs that are being used for simple queries OR we can do something faster and more efficient like this. Just because you can run a SotA model at "fast" speeds, again, severely misses the point.

And no, you can't run anything from Anthropic or OAI on-prem, so until you can there's really no comparison. If people want to continue down the path of gate-kept models with no other options then we'll all follow you off the cliff.

[0] https://taalas.com/products/

▲

Kirby64 an hour ago | parent [-]

Why are you representing this as such a binary here? For SLM we don’t need the Taalas stuff at all. Just run it locally on your own device if it’s truly a small model. And there’s plenty of larger models that can be run on-premise just fine.

I think it’s impressive that a frontier model can achieve 750t/s. That’s all. You can get similar insane token speeds from other open weight models too.

▲

windexh8er an hour ago | parent [-]

The irony here is, according to you, my take is the binary one. When your response is: well, we can all just run it on our devices - we don't need any other options!

You seem to be cool with a very small and gated ecosystem with whatever tech billionaires want you to have access to.

I grew up in the era where compute was diverse and open. You may think this is OK, but it's not. The more options we have and the more diversified they are the better tech will move back towards.

I'm not the one with the myopic view here. Enjoy your "on-device" models over in your utopia of a walled garden.

	▲	Kirby64 an hour ago \| parent [-]
		I think you’ve got things quite backwards if you think that the desire to run models on device or use any of the variety of open weight models (big or small) on premise is somehow bowing down to tech billionaires. Quite the opposite really. Once again, my statement is that the Taalas product is not a fair comparison because it runs an old outdated model. If you want to run a similar model at similar speeds (albeit not serially, but in parallel) you don’t need their product.