Remix.run Logo
YuechenLi 4 hours ago

Distilling frontier models is a brute force approach that rapidly hits diminishing returns after bootstrap because of the unevenness of the data. The simpler and more effective method is to have dedicated "teacher" frontier LLMs to generate targeted training data sets specifically for training new models and adjust on the fly based on feedback from the student model.