I thought one core result that led to LLMs was the realization that a specialized model is not necessarily better at a task than a general one.

That goes all the way back to at least to Stein's Paradox in 1955, sadly too few people get educated about Statistics and keep thinking specialized models will necessarily be better. If you want to estimate the batting averages of 3 MLB baseball players from samples, you are better off building a model to predict all of their batting averages than computing the mean from a sample of each one separately.

https://en.wikipedia.org/wiki/Stein%27s_example