> The models will need to partially specialize after hitting the hardware ceiling.

I'm incredibly skeptical of anything that goes "oh, we need to split AI into specialist systems".

I suspect that a lot of the delta between "our best models" and "base models they're trained on" lies in relatively low amounts of parameter changes driven by SFT and RL.

Model capacity is not the limit - the limit is our ability to drive those changes efficiently. Even things like "good research taste" could be represented as tiny in relation to total model capacity, if they could be trained into a model efficiently.

▲

tehjoker a day ago | parent [-]

At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.

I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.

▲

zer00eyz a day ago | parent | next [-]

> so parameter scaling is the primary way corporations are achieving increases in ability

Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.

We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.

	▲	ACCount39 18 hours ago \| parent [-]
		No, parameter scaling still works because it broadly increases sample efficiency. We don't have the raw data to make a Chinchilla optimal 100T, but that doesn't mean there are no gains in building up to 100T.

▲

ACCount39 18 hours ago | parent | prev [-]

I don't think "biology" and "elegantly" belongs in the same sentence. Let alone the one you wrote.

I suspect that the "elegant" workings of the human brain are a pile of hacks. Evolution crammed the brain full of patterns that are a good fit for the specific problems it typically solves - the way curve detectors and scale invariance are a good fit for CNNs.

We don't have millions of years of evolution of non-random init, but what we have is enormous datasets and compute at scale.

▲

tehjoker 12 hours ago | parent [-]

The brain is a pile of hacks, but it's still the most advanced thinking machine on the planet and it works on barely any power.

	▲	ACCount39 5 hours ago \| parent [-]
		LLMs are very competitive on power use at inference time. Sure, frontier LLMs require timeslots on massive server arrays. But they also get things done a lot faster than humans do. The net favors the LLM.