I have real world data here. On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context.

This is totally suitable (speed wise) for coding.

This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).

▲

ozgrakkurt 13 hours ago | parent [-]

How do you even fit that model in there? What quant is it? Are you using llama.cpp?

▲

notenchange 13 hours ago | parent [-]

No, I'm using strata. This is a MoE model, so it keeps the last used model in VRAM, and the rest in RAM. If it needs, it swaps the less used model in VRAM for the model in RAM for the next token.

Seems very slow, but it's actually fine.

	▲	notenchange 9 hours ago \| parent [-]
		P.S. Sorry, where you read “models” I meant experts.