| ▲ | saghm an hour ago | |
What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore) | ||
| ▲ | montroser 35 minutes ago | parent | next [-] | |
Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27B could not. | ||
| ▲ | msdz 36 minutes ago | parent | prev [-] | |
Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon). Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4. | ||