For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.

▲

throwawayffffas 43 minutes ago | parent | next [-]

If you are going down this path, MI350P and RTX PRO 6000 Blackwell is what you would want. A lot more compute a lot more vram bandwidth.

	▲	zozbot234 10 minutes ago \| parent [-]
		> A lot more compute a lot more vram bandwidth. But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.

▲

varispeed 2 hours ago | parent | prev [-]

But there are already plenty of products for most purposes.