Well absent a research breakthrough the only option is “scaling” which according to Sam Altman was logarithmic growth in model ability with number of weights.

Of course any stats undergrad could tell you this was a fairytale. Increasing number of weights only works until you exhaust the signal in the data. I’m not sure how he was allowed to get away with such a blatant lie but here we are.

This lie is effective because on a small time scale it’s impossible to tell the difference between logarithmic growth and logistic growth. If you maintain a fixed training data size, increasing the size of the model will get you logistic growth in model capability meaning that past a certain size you get effectively no gain in performance because you’ve already squeezed out 99% of the signal.

This disproves point number 1 in Sam’s thesis: https://blog.samaltman.com/three-observations

“The intelligence of an AI model roughly equals the log of the resources used to train and run it.”

He is playing loose with the language here because the only way this statement holds is when resources = breadth and depth of training data - not compute / model size.