We have probably hit a limit to scaling LLMs through raw parameter count alone, at least we're not seeing the exponential pace. I personally think we'll end up with a nice sigmoid curve plateauing in the sub 10T parameter regime. The amount of tokens processed (in inference) is increasing exponentially though (I've been following open router usage stats for years and it's always been exponential). We will of course make technological advances in hardware efficiency, and model parameter efficiency, but I think a much more plausible future is that VRAM needed for loading and serving individual models will slow down or even stop. We will need more chips, and more power, as demand continues to grow of course, but the operational lifetime of GPUs today will be a lot longer than the SoTA cards from 5 years ago.

> We have probably hit a limit to scaling LLMs through raw parameter count alone, at least we're not seeing the exponential pace.

Source?