My 5090 has a bandwidth of 1,792 GB/s, and that is just about good enough for basic (i.e. 27B models) local LLM inference.

The fastest M6 MacBook Pro (24/32GB configs) has a memory bandwidth of 170 GB/s. And the M5 Max is 614 GB/s.

Am I missing some nuance of the M6 MBP here, or am I right to assume it won't set the local LLM world on fire?

▲

quantumink 5 hours ago | parent | next [-]

The case of your 5090 having become a coveted artifact of great rarity in these times - may be a nuance of some measure.

Also, the following two have very different 'needs' to feel good enough:

1. heavy agentic workloads via harness vs 2. a simple private chat interface with a local instance

Even if an MBP's bandwidth were higher, it's tough to see a reason to run it as a constantly loaded 24/7 inference machine due to heat degradation.

P.S. still extremely awesome to be able to host and chat with a decently quantized 27B model on your laptop. ...and to be fair even a 32GB Air would do that! It is a thing of beauty: the amount of folks this may add to the hacker/tinker community.

▲

tedd4u 4 hours ago | parent | prev | next [-]

Isn't the laptop version of the 5090 is a more apples-to-apples comparison? It has 896 GB/s memory bandwidth (compared to the 614 on the M5 Max MBP).

▲

adolph 3 hours ago | parent | prev | next [-]

The 5090 does have a bandwidth advantage, but also must pass more data from main to GPU memory for any particular task. Comparisons of a 5090 to a DGX Spark at 273 GB/s are illustrative since the Spark uses unified memory similar to Apple's M-series. [0] The 5090 certainly wins most benchmarks but that doesn't mean the DGX Spark is not useful.

0. https://float16.cloud/en-en/ai-benchmark/dgx-spark-vs-rtx-50...

	▲	skohan an hour ago \| parent [-]
		Loading data to the GPU will generally never be the bottleneck. Yes that's a bit better for unified memory, but prefill and decode will still be bound by compute and memory bandwidth respectively. There's some value in having a huge amount of memory addressable by the GPU for loading larger models, but even that doesn't make sense for current gen models. For MOE models, newer inference engines with smarter GPU expert caching and pinned memory allow a GPU with lots of bandwidth to significantly outperform a Spark, so that leaves you with larger dense models that fit on a spark but not on a GPU where you might have a use-case. But even then it's hard. For instance a full-fat BF16 of a ~30B parameter model will fit on a 128GB spark and not a 32GB 5090, but at that size it's going to be frustratingly slow on the spark given the compute and bandwidth constraints. Imo these devices are mostly good for experimentation, or if you only want to do overnight jobs or training which wouldn't be possible on a GPU, and you don't want to pay $100k for a DGX station.

▲

lowbloodsugar 5 hours ago | parent | prev [-]

Is your 5090 in a laptop? o_0

▲

cbeach 5 hours ago | parent [-]

No. But does high memory bandwidth necessarily require desktop/server style power and cooling?

	▲	lowbloodsugar 3 hours ago \| parent [-]
		Laptop 5090 (24gb) is 800gb/s. M5 Max (128gb) is 600gb/s Rtx spark laptop (128gb) is 200gb/s. Desktop 5090 (32gb) is 1700gb/s. Desktop 6000 (96gb) is 1700gb/s and 600W. So, yes?