> ...we are already looking at dropping $100k on hardware to run local models...

Just think how much further that $100K would have gone if the hardware market wasn't so screwed-up.

Anecdote: I priced-out adding 1TB of RAM to a four node cluster a couple months ago. The cluster was purchased in fall of 2024 w/ 4 nodes, each with 256GB RAM. The nodes cost just over $14K apiece back in 2024 (entire box, not just the RAM).

Dell wanted >$90K a couple months ago to add 256GB to each node.

▲

cyberax an hour ago | parent [-]

> Dell wanted >$90K a couple months ago to add 256GB to each node.

RAM is expensive, but not THAT expensive. I just bought 128Gb for about $5k for our build cluster (it's not even for AI, sigh). Even if you need larger-sized DIMM sticks, it's still going to be in the vicinity of ~15k tops.

	▲	EvanAnderson 14 minutes ago \| parent \| next [-]
		It was crazy. I found the part on the open market for a lot less but the edict from the Customer was to buy from Dell to keep the support entitlement intact. That inflated the price to an astronomical level to be sure. I haven't had problems w/ Dell support and 3rd party memory, personally, but given the machines' application I understood the concern.
	▲	an hour ago \| parent \| prev [-]
		[deleted]