Does vast amounts of lower and lower quality data help much? If you can train on the entire feeds of social media, you keep up on recent pop culture trends, but does it really make LLMs much smarter?

Recent progress on useful LLMs seems to involve slimming them down.[1] Does your customer-support LLM really need a petabyte of training data? Yes, now it can discuss everything from Kant to the latest Taylor Swift concert lineup. It probably just needs enough of that to make small talk, plus comprehensive data on your own products.

The future of business LLMs probably fits in a 1U server.

[1] https://mljourney.com/top-10-smallest-llm-to-run-locally/

I think this is the right question to ask. I think it depends on the task. For example, if you want to predict whether someone has cancer, then access to avast amounts of medical information would be important.