This pattern is playing out a lot. Running LLM inference at scale on apps typically lacks ROI. But I increasingly see folks use LLMs to help improve classical techniques.

So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.

Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.