Is it me or is that not what that graph shows? That line has ~no relation to the datapoints it's purporting to be a fit for.

▲

kaarelson 20 hours ago | parent | next [-]

Correct, the line is not fitted to n=27 data points. It simply connects the highest params size between 2023 and 2026. To clarify, I wouldn't expect traditional research groups to pre-train anything >3B params WM, because this can't be done on a single GPU anymore. Just wanted to ship this as quickly as possible. Happy to do a follow-up with an actual log-scale linear regression, though.

	▲	kadoban 20 hours ago \| parent [-]
		It almost would feel like a fair graph, but it's also below that one ~outlier in the middle. So it's ignoring all data but two points and purporting to fit an exponential to it. That's within spitting distance of meaningless, right? There's an infinite number of exponentials that fit the same two points just as well, and way more non-exponentials too.

▲

fithisux a day ago | parent | prev [-]

True.