Remix.run Logo
elliotbnvl 6 hours ago

$500k is a vast overestimation. For massive concurrency at FP8 or even BF16 maybe.

NVFP4 at reasonable speeds (~120 tok/s) and concurrency is possible at a $80/90k figure with today's prices, maybe even less. That buys you 6 RTX 6000 PRO Blackwells, a decent CPU and motherboard, power supply. 576gb of VRAM.

You could do it for under $50k if you're OK with 40 tok/s decode, ~1200 tok/s prefill.

hbbio 4 hours ago | parent | next [-]

Yes, a single GB300 workstation also does it, probably even more than 120tok/s.

Official price 85k...

__m 5 hours ago | parent | prev | next [-]

How fast will the hardware become outdated? Are there big improvements expected in the next 3 years?

Tepix 25 minutes ago | parent | next [-]

I think there is a gap right now for running large models such as GLM 5.2 in Q4 or Q8. My hope is on Intel Crescent Island 480GB cards. Let‘s see how expensive they‘ll be.

easygenes 4 hours ago | parent | prev | next [-]

M5 Ultra will ship before end of year, likely. Though with current RAM shortage, likely max spec will be 256GB and in short supply.

In late 2027 or early 2028, Nvidia will release Vera Rubin DGX Spark, likely with double or better the performance of current Blackwell, though unclear if memory capacity will go up much from current 128GB. Two to four of those will run models like this decently.

In 2028 we should expect Vera Rubin RTX discrete lineup, including the replacement to the RTX PRO 6000. Likely memory spec will be minimum 128GB. Good chance of up to 200GB. Two to four of those will run NVFP4 models in this class very well.

jiqiren an hour ago | parent | next [-]

I hope all this speculation comes true. Right now this ram crunch is ridiculous.

4 hours ago | parent | prev [-]
[deleted]
digitaltrees 3 hours ago | parent | prev | next [-]

I feel like the models are good enough for a decade of future work. So Once you have a working set up you can keep using it to do the work at the same level. There will be better stuff and may make that type of work obsolete but if you can do useful things it won’t be worth less.

segmondy 3 hours ago | parent | prev [-]

P40 was release 2016 and still selling like hotcakes!

easygenes 4 hours ago | parent | prev [-]

[dead]