Remix.run Logo
▲ vineyardmike an hour ago

According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.

▲syntaxing an hour ago | parent [-]

This has been observed by the broad community for llama cpp.

▲nicce an hour ago | parent [-]

llama.cpp is one of the slowest these days

▲saghm an hour ago | parent [-]

What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)

▲montroser 34 minutes ago | parent | next [-]

Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27B could not.

[1]: https://github.com/iamwavecut/ninfer-all

▲msdz 36 minutes ago | parent | prev [-]

Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.