256gbs memory bandwidth is about 1/4 that of a 3090. It would be a better buy with half the memory at 4x the speed.

Are you sure about that? High memory speed is great for dense models, or when serving at high concurrency.

However for local single-user setups, it's often better to have access to more capable/bigger MoE models at reasonable speeds and lower concurrences, which is enabled by these platforms.

▲

roadside_picnic 2 hours ago | parent [-]

If you're using a MoE model, then why do you care about the larger RAM offered by these devices? That's the main problem with low bandwidth devices: they limit the effective ram you can make use.

I do (and have historically done) quite a work with both local LLMs and local diffusion models. I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s

While the M3 Max MBP can serve up MoE reasonably fast (~60 token/sec)the RTX 4090 is an entirely different experience (~170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX.

The raison d'etre for the DGX spark is not practical home inference, but rather offering the same fundamental architecture as data center cards for a affordable CUDA prototyping. If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. The DGX spark solves this by having the same fundamental setup as what those cards have.

That makes these non-NVIDIA DGX-like devices confusing to me. The entire benefit of the DGX series is the NVIDIA architecture itself.

Anyone interested in home LLMs should decide whether a Mac or a dedicated GPU is the more sensible path based on their budget and other computer use. Each has their own benefits.

	▲	wolttam an hour ago \| parent [-]
		I run DSv4 Flash at home on 2 DGX Sparks and am pretty sure there is no more cost effective way for me to do so. I'm not interested in running smaller models.

▲

StableAlkyne an hour ago | parent | prev | next [-]

Any performance gains caused by the internal bandwidth of the card will evaporate once you spill into system RAM, because now your bottleneck is probably a slow PCI lane.

And if your jobs do fit onto a 24GB card, then you are not the target user for the "AI mini PC" niche that these guys are trying to carve out

▲

muyuu 2 hours ago | parent | prev | next [-]

it depends

it allows you to run smaller models much better

imo 3090s make the most sense if you can buy at least 2x ideally 4x but of course we're talking about a completely different budget at that point

▲

cyanydeez 2 hours ago | parent | prev [-]

what matters is how much memory it has; with the new MTP models, Qwen3.6 with 35B MOE, it's pumping out tokens up to ~80k context with little slow down.

It's great to get lots of tokens, but being able to handle and extent context is why it'll continue to be a great machine compared to any of the small graphics cards.