I just ran an experiment, I took the banking77 dataset, took the labels out, and got an LLM (Opus 5) to label them, then trained a logistic regression classifier (Jeffy http://github.com/nicobrenner/jeffy) on the resulting set

Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)

But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification