This model I think will be too slow for that on Spark, even at 4 bit quant.

It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited

I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size:

NVFP4 Q/K/V/O and MLP projections: ~13 GB/token

BF16 attention gates: ~3 GB/token

BF16 LM head: ~2.5 GB/token

Total: ~18.9 GB/token

At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.

▲

rao-v 2 hours ago | parent [-]

Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.

	▲	cmrdporcupine 14 minutes ago \| parent [-]
		You're right. I'm getting ~33tok/sec w/ dflash on it, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based) That's pretty respectable. Still working on optimizing and cleaning up before I push it.