> ChatGPT was the turning point of realizing we needed just more compute
Ilya Sutskever has been saying more compute = better perf since at least 2012.
... the real conclusion now is that if you have a large big data set and you train a very large deep neural network success is guaranteed ...
https://www.youtube.com/watch?v=-uyXE7dY5H0&t=1185 In the end, the network's size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate ... All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets ...
https://proceedings.neurips.cc/paper_files/paper/2012/file/c...