Remix.run Logo
▲ zer00eyz a day ago

> so parameter scaling is the primary way corporations are achieving increases in ability

Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.

We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.

▲ACCount39 a day ago | parent [-]

No, parameter scaling still works because it broadly increases sample efficiency. We don't have the raw data to make a Chinchilla optimal 100T, but that doesn't mean there are no gains in building up to 100T.