I feel like these types of benchmarks mostly test what the mainstream tells you is possible and or works. You rarely get to hear the different implementations of inference engines, different types of setups (e.g. hybrid ones or older hardware) and their pros and cons. Also it seems these were tested with very minimal token usage.

Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.

If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.

▲

PaulKeeble an hour ago | parent | next [-]

Locally the MOE based models get a lot more out of the hardware than a dense model. Qwen 3.8-27B is the current golden child if you have >24GB VRAM but because its a dense model it is pretty slow to run and requires a lot of hardware. A lot of the guides are very much focussed on getting the dense models to run but in practice the MOE models with few active parameters are just faster and easier to run without too much loss of intelligence, they behave a lot more like their peak size than the active parameter size.

RAM is going to be the big bottleneck for MOE (at least for now) as it will determine the quantisations you could run.

On a 3080 10GB and 32GB of RAM I can run the qwen-next coder model and that is a 127B model and it gets 40-50 Token/s with 65k context. That is a lot more usable than a Quantised Qwen3.8-27B in my experience both because its faster and more capable.

I am looking forward to Qwen4 soon, I am hoping we get some more 27-35B MOE models, those are going to be really good for local hardware.

▲

samuell 3 hours ago | parent | prev | next [-]

Thanks for sharing! I would love for some go to source for this type of insider info from the people who actually have tried. (The best I've found so far are some guys on Twitter/X like Alex Fateev (https://x.com/superalesha) and AJ (https://x.com/ItsmeAjayKV))

I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.

At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.

▲

Abishek_Muthian 2 hours ago | parent [-]

Try to use your IGPU as well if you have enough RAM, almost everyone has dual GPUs in their computers.

Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.

Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.

I might write more about this setup soon.

	▲	samuell an hour ago \| parent [-]
		Interesting advice, gotta look into that, thank you!

▲

testerius 2 hours ago | parent | prev | next [-]

It sounds great and I hope local LLM/AI will be more powerful and available even on old hardware.

I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.

Is it even worth for trying something on my GPU? What do you think?

And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.

▲

lopotubez an hour ago | parent [-]

Models and inference engines are getting better and more optimized everyday so it is definitely worth experimenting with your GPU. You should probably check on newer finetuned and or REAP models Huggingface that could fit on your system.

I believe in your case the model quality matters a lot. To my (trust me bro) experience any MOE model, finetuned or not, with params < 19B even with Q4-quant always give lackluster results. So any Qwen2.5 wouldn't suffice to even simple coding tasks. For your case maybe you can bump it up to a Gemma4 E2B/E4B or Qwen3.5-9B. Also make sure that your current build of llamacpp actually uses your GPU with CUDA.

	▲	testerius 32 minutes ago \| parent [-]
		Yes, I was using CUDA toolkit on my Debian. Anyway, thanks for reply. I think I could try something but definitely it looks like it will not have better quality, even for writing simple code function by function :P

▲

OtomotO 2 hours ago | parent | prev [-]

I need to play around with this.

I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.

I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.

I must be doing something wrong.

Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".

llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder

using it in opencode

{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }

▲

lopotubez 39 minutes ago | parent | next [-]

Your setup is definitely a monster compared to mine so it should give much better results. I also believe you run dense models with decent results with that setup.

Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):

[*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2

[qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2

I am also certain that these flags could be improved somehow depending on your system.

▲

PaulKeeble an hour ago | parent | prev | next [-]

Ideally you want to find something that allows a context window for at least 65k for agent work. Over 100k is better but 32k is about the bare minimum for anything in a harness/agent like Pi. You might have to drop to a lower quantisation to make that possible but context is the key for longer lots of thinking questions like this.

If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.

So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.

▲

trucks-refinish an hour ago | parent | prev [-]

I would give it significantly more context, if you don't have enough room you can quantize the kv cache with some flags that I don't remember ottomh.

I would go ahead and give it at least 64k but the more the better really!

I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.

	▲	PaulKeeble an hour ago \| parent [-]
		--cache-type-k q8_0 --cache-type-v q8_0 would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.