Leaks from within OpenAI have made it pretty clear that they've been struggling to achieve significant improvements lately by simply scaling up parameter size. Experts like LeCunn have also been vocal that blindly scaling up is a dead end.

(Incidentally, the line of skill improvement isn't "exponential". It's been incremental in improvements per generation, but generations have been coming thick and fast of late, and have grown in parameter count exponentially since 2017.)

Speaking more broadly, LLMs don't have to "hit a wall" in scaling to become uneconomical. If incremental improvement continues to come at exponential cost, however, then the fundamental value argument falls apart.

Setting all that aside, even presuming that model performance scales linearly with dimensionality, there are just fundamental limits to the size of the training corpuses. Quality training data is not unbounded and infinite. Given the same size corpus of training data, there's a hard theoretical limit to how much meaning and inference a model can wring out of it.

And then there are other issues with the whole business model. For one thing, it's predicated on continual full scale retraining to achieve even modest gains in skill and relevancy. Topical and targeted learning requires a full retraining. Etc cetera.

I think that the next generation of AI will lean more heavily on RL to be useful beyond a few months. I also think that the energy requirements of a particular technology are a good proxy to whether it's got a realistic future.