| ▲ | syntaxing an hour ago | ||||||||||||||||||||||||||||||||||||||||
I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel. | |||||||||||||||||||||||||||||||||||||||||
| ▲ | Aurornis 43 minutes ago | parent | next [-] | ||||||||||||||||||||||||||||||||||||||||
It's less than 24 hours old. The inference engines haven't been optimized for it yet. | |||||||||||||||||||||||||||||||||||||||||
| ▲ | flatline 16 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||
I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate. | |||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||
| ▲ | vineyardmike an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||
According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster. | |||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||
| ▲ | baby 12 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||
Can you autoresearch optimize it? | |||||||||||||||||||||||||||||||||||||||||
| ▲ | woadwarrior01 an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||
The NV in NVFP4 stands for NVIDIA. :) | |||||||||||||||||||||||||||||||||||||||||
| ▲ | plasticchris an hour ago | parent | prev [-] | ||||||||||||||||||||||||||||||||||||||||
It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes. | |||||||||||||||||||||||||||||||||||||||||