At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.

I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.

> so parameter scaling is the primary way corporations are achieving increases in ability

Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.

We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.

I don't think "biology" and "elegantly" belongs in the same sentence. Let alone the one you wrote.

I suspect that the "elegant" workings of the human brain are a pile of hacks. Evolution crammed the brain full of patterns that are a good fit for the specific problems it typically solves - the way curve detectors and scale invariance are a good fit for CNNs.

We don't have millions of years of evolution of non-random init, but what we have is enormous datasets and compute at scale.