Remix.run Logo
▲ syntaxing an hour ago

I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.

▲Aurornis 43 minutes ago | parent | next [-]

It's less than 24 hours old. The inference engines haven't been optimized for it yet.

▲flatline 16 minutes ago | parent | prev | next [-]

I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.

▲6_7 8 minutes ago | parent [-]

I'm so sorry you're getting how many tok/s?

▲vineyardmike an hour ago | parent | prev | next [-]

According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.

▲syntaxing an hour ago | parent [-]

This has been observed by the broad community for llama cpp.

▲nicce an hour ago | parent [-]

llama.cpp is one of the slowest these days

▲saghm an hour ago | parent [-]

What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)

▲montroser 32 minutes ago | parent | next [-]

Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27B could not.

[1]: https://github.com/iamwavecut/ninfer-all

▲msdz 34 minutes ago | parent | prev [-]

Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.

▲baby 12 minutes ago | parent | prev | next [-]

Can you autoresearch optimize it?

▲woadwarrior01 an hour ago | parent | prev | next [-]

The NV in NVFP4 stands for NVIDIA. :)

▲plasticchris an hour ago | parent | prev [-]

It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.