> Per David Abitbol, who leads AI ops there: 56 Nvidia B200s, a blended 30 cents per million tokens, roughly a tenth of what they paid Anthropic and OpenAI.

So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.

We'll see if that's a trend for mid-big companies or just an experiment.

All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.

It is in the article: "the gains showed up when tokens stopped being rationed" - so it is not just a "cost per token before/after" calculation. Once you told your employees "token are free, flatrate for all - go nuts!" they started to use it for all sorts of things and increased productivity. That is not easy to calculate.

I doubt local will be all that common aside from regulatory requirements but buying them off your cloud provider will be the new norm for many enterprises.

The slice of the pie where frontier models are required is just gonna be smaller and smaller. And at point where AI wrote docs, tests, and all the easy and medium difficulty functions you are already saving a ton per employee

> So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling).

Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.

Unless you have very strict requirements, it does not make any sense to invest in hardware. Most companies just rent it. Also, it's pretty hard to get the hardware.

The B200 was launched in 2024 if I'm not mistaken. The big gamble with this might be how fast these will deprecate in value/use as soon as Nvidia launches a new and better product.

At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?

Personally, I think Anthropic and OpenAI will introduce enterprise plans such as $2,000/month/engineer plans that will give 100x more tokens than the current max plans and then charge API pricing.

Yes, margins will go down a bit but the frontier labs still have some leverage.