Ways to think about token pricing

mips_avatar an hour ago | parent | next [-]

I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.

▲

Analemma_ 33 minutes ago | parent | next [-]

You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.

▲

Legend2440 22 minutes ago | parent [-]

DRAM prices have been highly cyclical for a long time. They'll come down again.

	▲	draygonia 13 minutes ago \| parent [-]
		From what I'm reading on the shortage, they are very aware of this fact (prices being highly cyclical) and they're trying to avoid overproducing RAM to avoid the oversupply glut. That's probably going to hurt the chance of prices coming down anytime soon.

▲

cyanydeez an hour ago | parent | prev [-]

sure, if there werent capitalists digging their moat. in 1 year local inferwnce doubled cost from ram alone because cloud vendors cornered markets on their circular cash flow expectations.

	▲	mips_avatar an hour ago \| parent [-]
		Yeah but Samsung/Micron/SK hynix are building new fabs, and Chinese Ram will be able to fully supply the lower end ddr5 ram market within 2 years. The crunch is temporary and on the other side of it there will be incredible hardware at the $10-20k price point.

▲

nekusar 2 hours ago | parent | prev [-]

1. Slot machine

You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.

And it gets tantalizingly close, but for 100% gotta pull the arm again!