The 5090 does have a bandwidth advantage, but also must pass more data from main to GPU memory for any particular task. Comparisons of a 5090 to a DGX Spark at 273 GB/s are illustrative since the Spark uses unified memory similar to Apple's M-series. [0] The 5090 certainly wins most benchmarks but that doesn't mean the DGX Spark is not useful.

0. https://float16.cloud/en-en/ai-benchmark/dgx-spark-vs-rtx-50...

Loading data to the GPU will generally never be the bottleneck. Yes that's a bit better for unified memory, but prefill and decode will still be bound by compute and memory bandwidth respectively.

There's some value in having a huge amount of memory addressable by the GPU for loading larger models, but even that doesn't make sense for current gen models. For MOE models, newer inference engines with smarter GPU expert caching and pinned memory allow a GPU with lots of bandwidth to significantly outperform a Spark, so that leaves you with larger dense models that fit on a spark but not on a GPU where you might have a use-case.

But even then it's hard. For instance a full-fat BF16 of a ~30B parameter model will fit on a 128GB spark and not a 32GB 5090, but at that size it's going to be frustratingly slow on the spark given the compute and bandwidth constraints.

Imo these devices are mostly good for experimentation, or if you only want to do overnight jobs or training which wouldn't be possible on a GPU, and you don't want to pay $100k for a DGX station.