For the ones who are still wondering how so much profits, inference is a highly profitable domain: https://inferencex.semianalysis.com/profit-estimator

I think we need to collectively get a grip on statements like this.

There is no reason to think inference can or will remain profitable even if you assume pure-play inference providers who don't do any model training. Serving an LLM isn't child's play but it's also not so hard as to count as a moat. Inferencing is:

1. Selling a largely undifferentiated product.

2. Into a highly competitive market.

3. To customers who are primarily sensitive to price.

Industries like this have always existed and in the modern era tend to have margins of about 3% to 5%. Any level of margin higher than that gets competed into the ground.

If you want to claim inference is highly profitable in general, you're going to have to answer why it won't end up like supermarkets or airlines, both of which have bigger moats than inference.

The other problem with this analysis is obviously that you can't just stop training models. Even if you don't care about improving capability or performance, the knowledge in their weights gets stale. Cost of inferencing has to take into account the ongoing cost of training, you can't cleanly split one from the other.

Accounting for initial acquisition cost and typical depreciation/failure rate? I assume they already consider colo price, electricity, etc

Fascinating site. Thanks for linking. However note if you change to "open router" pricing in some cases even B200 which is ~2y old has negative margin (and AFAIK ppl are using longer amortization than 2y - I do not know how cost are calculated here though).

Why isn't ordered by profit % per GPU? It' surely the most important indicator