I think the people that are saying that mean that the inference is massively profitable. From what I've seen, the margins are between 50 and 90% on an API call. It's the training that's very, very expensive!

As far as I understand, for both OpenAI and Anthropic, currently the majority of their GPUs are being used for training, with only a smaller portion being used for actual customer serving.

Inference requires training. That's a disingenuous point.

It's similar to people talking about drug companies charging hundreds of dollars for a medication that costs dollars to manufacture. Nevermind that it took them 15 years of research and trials to discover it and get it through trials and the approval process. Nevermind again all the drugs that failed at one of those steps.