You don't have to restrict yourself to that, you can create synthetic data or just train on both kinds of data.

I still don't understand what the bitter lesson has to do with this. First of all, it's only a piece of writing, not dogma, and second of all it concerns itself with algorithms and model structure itself, increasing the amount of data available to train on does not conflict with it.