Remix.run Logo
▲ bobmarleybiceps a day ago

I think a lot of model architecture changes don't move the needle ~that much and are basically done to improve runtime performance and scaling. - There's tons of cases of ml researchers, tech companies, and labs that propose new and improved models that turn out to not really be significant quality improvements (if at all) once the community starts experimenting with them.

it does seem like (IMO) the biggest leaps have basically been perf & scaling + RL on high quality data + harnesses.

▲versteegen a day ago | parent [-]

Yes, but "improving runtime performance and scaling": does directly translate into huge quality improvement too, because then you can you a much larger or deeper model, train it for much longer, and have a larger context.

▲bobmarleybiceps 39 minutes ago | parent [-]

yeah I mostly mean research papers where people are like "we add a new 'ultra lightning n-th order functionaliser quarternionized metric space-enabled infinite dimensional Fourier-routed experts' that improve results by 0.01% on our benchmarks and chosen baselines (we totally didn't spend way longer tuning our own model than our comparisons)"