It’s a fair perspective that you’re bringing, but what comes to mind for me in response is that I have a mental model of two types of machine learning inferences that provide value:

Type B: “directionally correct” ML inferences that provide value by virtue of outputting an inference regardless of its accuracy. The value comes from providing automated decision that allows someone or something to move forward in a process, similar to management consultants. This is used heavily in ad tech and (social) media algorithms. I would even put gambling and investing inferences in this bucket as the accuracy of algorithms isn’t really required to be that high to provide alpha.

Type A: ML inferences that are creating or replacing systems where high accuracy is table stakes. This is applications in industrial engineering, chemical engineering, biotechnology, etc. If you can’t beat any existing automated or non-automated systems in terms of accuracy, your product is dead in the water. You can make it a few years by faking it but eventually the pied piper makes his call, and a major contract falls through. This has always been the lifecycle of products I’ve been on. It’s an incredibly difficult problem. “Faking it” here means piling on heuristics upon heuristics to convince someone to sign a contract, even if the underlying precision-recall tradeoff curve isn’t financially viable. This is the basis for the conversations between myself and my colleagues that a model that can’t provide actionable insights without a wall of heuristics / post-processing is a red flag for a product.

Anyway, that generalizes to LLMs because LLMs can provide so much value in the Type B category, but anyone promising Type A products based on LLM technology is treading on thin ice, imo.

Edit: also, as an aside, feature extraction and post-processing of inferences is necessary for any model, but if that code becomes bloated is when I lose confidence in the model’s ability to provide inference value.

There’s a classic stack overflow from a decade ago that I remember where a ML engineer / DS was asking if a linear regression model trained on sorted data (X, Y sorted independently) had any inference value. They asked because their manager insisted that was the correct way to do it because it minimized training error. Well, duh. Two sorted lists of values are maximally correlated.

This is an extreme example of what I mean by “preprocessing data”.

An extreme example of “postprocessing results” would be to create a decision tree based on the output of a decision tree algorithm that re-sorts values into something that is palatable for the customer, based on feedback from the customer on a case-by-case basis. “Hey this was a false positive…” -> add an “if” statement to the postprocessing.