| ▲ | bobmarleybiceps a day ago | |||||||
I think a lot of model architecture changes don't move the needle ~that much and are basically done to improve runtime performance and scaling. - There's tons of cases of ml researchers, tech companies, and labs that propose new and improved models that turn out to not really be significant quality improvements (if at all) once the community starts experimenting with them. it does seem like (IMO) the biggest leaps have basically been perf & scaling + RL on high quality data + harnesses. | ||||||||
| ▲ | versteegen a day ago | parent [-] | |||||||
Yes, but "improving runtime performance and scaling": does directly translate into huge quality improvement too, because then you can you a much larger or deeper model, train it for much longer, and have a larger context. | ||||||||
| ||||||||