SVMs in big data were already well liked back in 2010 as they could be scaled via map-reduce.

An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.

So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.

LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.

I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.

Agree to all of this but I would make a minor correction: the interpretability is true under certain conditions, e.g., you use something like n-grams and train an SVM with a linear kernel. You could change the representation back then too, e.g., LDA (topic models), Latent Semantic Indexing, and of course, you could change the SVMs kernel as well, e.g., use a sequence kernel [1]. Some of these were much less interpretable. But the common pipeline back then was indeed n-grams + Linear SVM. Today it is easy to modernize the pipeline by just changing the representation step to use an embedder like ModernBERT. I have a "one page" classifier if you want to try it out - [2].

[1] I went down the rabbit hole of using sequence kernels quite a bit back then, see https://arxiv.org/abs/1910.13800, but the timing was terrible. RNNs and variants like LSTMs were beginning to become popular, and everyone (including me) switched.

[2] https://pocket-text-lab.abhishek-ghose-82.workers.dev/

LLMs, and other transformer models before them like BERT for example, have the benefit of "transfer learning."

If you want to make a ticket classification model, you don't need to train across many customers doing ticket classification, who might not want you to train on their data and whose data is impossible to obtain unless you already are a large company. Instead, you can train on a completely different task which forces the model to learn the syntactic and semantic structure of the English language — say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English — and then fine-tune that model on your specific task with much less data.

Because the initial task is so general, you can have one company do the expensive pretraining, share the base model, and then have everybody else just do the finetuning on their (small) proprietary dataset.

LLMs and models like Jev go one step further, letting you use them as classifiers with no training at all, with nothing but rules in the prompt. And with a few tricks, you can indeed use them to get pretty high-quality labels, which you can then distill into a fine-tuning dataset for a much smaller model.

And you can now get that world knowledge out of llms and generate synthetic data to train your svm and fine tune on your own dataset!