Remix.run Logo
▲ GodelNumbering 2 hours ago

If you take the argument at a face value and follow, this also implies that any advances are largely correlated with base model sizes (note the latest drop is from Astra, not sol 6x). And importantly, to drop the log perplexity incrementally, you have to increase the base model sizes continuously and likely disproportionately.

In other words, there is only so far you can go with the model of a given size before needing to increase it for smaller incremental gains.