> You speak so authoritatively about quality and performance of these models, yet there are no quantitative metrics that correlate to real world outcomes that indicate that the quality and performance of these models is anything but subjective noise and classic benchmark nonsense.

I'm responsible for AI roll out at a small business and we've had data science go over these things internally in terms of what results we get for 12+ months now. Its just my experience that is roughly the results we've seen using Deepseek, etc. and comparing cost/results vs. Anthropic/ChatGPT.

> A company consumed half a billion dollars worth of tokens in a month and nobody noticed anything until the bill came due.

It was sourced from one anonymous source. Its highly unlikely to be true in my view, but hey, you do you.