What I think would be perfect is a model that could run on a single DGX spark and be competitive with DSV4 Flash 731. Flash is already a game changer. Hopefully meta plans on this, like the old 70b. V4 flash is smart enough for any use but slightly too big. 27b-30b isn’t intelligent enough.

▲

cmrdporcupine 3 hours ago | parent | next [-]

This model I think will be too slow for that on Spark, even at 4 bit quant.

It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited

I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size:

NVFP4 Q/K/V/O and MLP projections: ~13 GB/token

BF16 attention gates: ~3 GB/token

BF16 LM head: ~2.5 GB/token

Total: ~18.9 GB/token

At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.

▲

rao-v 2 hours ago | parent [-]

Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.

	▲	cmrdporcupine 14 minutes ago \| parent [-]
		You're right. I'm getting ~33tok/sec w/ dflash on it, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based) That's pretty respectable. Still working on optimizing and cleaning up before I push it.

▲

127 3 hours ago | parent | prev [-]

DSV4 Flash 0731 already runs on RTX 4090 24GB + 128GB system RAM at a usable tok/s and quantization.

▲

Gecko4072 3 hours ago | parent [-]

You personally? Just curious. Context window is also a factor and ram isn’t really cheap. Sparks are assembled units which I like.

▲

dannyw 2 hours ago | parent [-]

For the same price as a DGX Spark here (A$8499) I can buy roughly 544GB of DDR5-5200MHz from retail; which on a quad channel platform would deliver ~160gb/s real world; and ~320gb/s with octa channels (Xeon, Threadripper Pro).

If you can afford it or somehow find a used unit, you can go Epyc for 12 channels.

8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world.

With some optimisation and maths, it’s entirely plausible to ach

You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still.

Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet.

Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.

	▲	danielEM 6 minutes ago \| parent [-]
		Been investigating these multichannel AMD based platforms last year and seem like none of them can in real scenarios utilize anywhere close to their theoretical bandwidth.