Isn't this just the Bitter Lesson over and over again? Specialised approaches get outperformed by more compute and a simpler general approach pretty much every time.

Bitter lesson is wrong. Large general models are outperformed by small specialized models.