Aren't these specs already outdated? Seems meh. I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.

Give me 1TB RAM and 5TB/s or something that exciting.

▲

entrope 4 hours ago | parent | next [-]

For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.

▲

throwawayffffas 43 minutes ago | parent | next [-]

If you are going down this path, MI350P and RTX PRO 6000 Blackwell is what you would want. A lot more compute a lot more vram bandwidth.

	▲	zozbot234 10 minutes ago \| parent [-]
		> A lot more compute a lot more vram bandwidth. But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.

▲

varispeed 2 hours ago | parent | prev [-]

But there are already plenty of products for most purposes.

▲

throwawayffffas an hour ago | parent | prev | next [-]

This is all the CPU companies playing catch up with Nvidia and AMD.

The thing you are describing is much closer to AMD and Nvidia latest generation server offerings.

See MI455X for the absolute cutting edge and MI350P for something you can use outside of a datacenter.

▲

tempest_ 4 hours ago | parent | prev | next [-]

5 year old GPUs are still at crazy prices.

Plenty of businesses want in local inference and are priced out.

▲

varispeed 2 hours ago | parent [-]

You can't do quality inference on such a toy. It's a myth.

	▲	redanddead 2 hours ago \| parent [-]
		A great opportunity for huawei Nobody helps China like American corps helps China. Domestic anticompetitive behavior spurs demand for international competition

▲

mciancia 3 hours ago | parent | prev [-]

8 of those cards give you 1TB of ram and 4.4TB/s

▲

PhunkyPhil 3 hours ago | parent [-]

It may be a semantic thing but I don't think bandwidth pooling works like that? Unless your blocks are distributed perfectly between cards there's going to be data passed between each card and from VRAM to compute. Correct me if I'm wrong though

	▲	throwawayffffas 33 minutes ago \| parent \| next [-]
		It depends on how you slice the weights. You can split along layers which means each GPU waits for the previous one to finish or you can split the tensors across gpus. Splitting the tensors is closer to n times the effective vram bandwidth but there are still overhead, sync waits, and reduce steps that eat away at the overall performance.
	▲	msandford 3 hours ago \| parent \| prev [-]
		I think the way it works is that the weights have to get streamed through the compute and the weights can be split very easily. With multiple cards you do have a coordination/communication phase after each layer of weights but that's usually much less significant than the compute phase.