From my understanding, the field of AI was all about knowledge representation. Researchers in the past handcrafted representations with expert knowledge but that only gets you so far. So instead, why not learn representations from data directly?

That's the current stage we're at and is the whole scraping the entirety of human knowledge thing. Compute has gotten good enough and data readily accessible to do all this, plus we have architectures like transformers that scale really nicely.