What about AMD Ryzen AI Max+ 395 Mini PCs with upto 128GB unified memory?

Their memory bandwidth is the problem. 256 GB/s is really, really slow for LLMs.

Seems like at the consumer hardware level you just have to pick your poison or what one factor you care about most. Macs with a Max or Ultra chip can have good memory bandwidth but low compute, but also ultra low power consumption. Discrete GPUs have great compute and bandwidth but low to middling VRAM, and high costs and power consumption. The unified memory PCs like the Ryzen AI Max and the Nvidia DGX deliver middling compute, higher VRAMs, and terrible memory bandwidth.

▲

MalikTerm 5 days ago | parent | next [-]

It's an underwhelming product in an annoying market segment, but 256GB/s really isn't that bad when you look at the competition. 150GB/s from hex channel DDR4, 200GB/s from quad channel DDR5, or around 256GB/s from Nvidia Digits or M Pro (that you can't get in the 128GB range). For context it's about what low-mid range GPUs provide, and 2.5-5x the bandwidth of the 50/100 GB/s memory that most people currently have.

If you're going with a Mac Studio Max you're going to be paying twice the price for twice the memory bandwidth, but the kicker is you'll be getting the same amount of compute as the AMD AI chips have which is going to be comparable to a low-mid range GPU. Even midrange GPUs like the RX 6800 or RTX 3060 are going to have 2x the compute. When the M1 chips first came out people were getting seriously bad prompt processing performance to the point that it was a legitimate consideration to make before purchase, and this was back when local models could barely manage 16k of context. If money wasn't a consideration and you decided to get the best possible Mac Studio Ultra, 800GB/s won't feel like a significant upgrade when it still takes 1 minute to process every 80k of uncached context that you'll absolutely be using on 1m context models.

▲

codedokode 5 days ago | parent | prev [-]

But for matrix multiplication, isn't compute more important, as there are N³ multiplications but just N² numbers in a matrix?

Also I don't think power consumption is important for AI. Typically you do AI at home or in the office where there is lot of electricity.

	▲	evilduck 5 days ago \| parent [-]
		>But for matrix multiplication, isn't compute more important, as there are N³ multiplications but just N² numbers in a matrix? Being able to quickly calculate a dumb or unreliable result because you're VRAM starved is not very useful for most scenarios. To run capable models you need VRAM, so high VRAM and lower compute is usually more useful than the inverse (a lot of both is even better, but you need a lot of money and power for that). Even in this post with four RPis, the Qwen3 30 A3B is still an MOE model and not a dense model. It runs fast with only 3B active parameters and can be parallelized across computers but it's much less capable than a dense 30B model running on a single GPU. > Also I don't think power consumption is important for AI. Typically you do AI at home or in the office where there is lot of electricity. Depends on what scale you're discussing. If you want to get similar VRAM as a 512GB Mac Studio Ultra with a bunch of Nvidia GPUs like RTX 3090 cards you're not going to be able to run that on a typical American 15 AMP circuits, you'll trip a breaker half way there.

▲

ekianjo 5 days ago | parent | prev [-]

Works very well and very fast with this Qwen3 30B A3B model.