| ▲ | euleriancon 2 hours ago | |
This isn't as easy as it sounds. Every ML model is struggling to balance between generalization and test performance. Taking a good model like GLM5.2 and just fine tuning it on coding can decrease real world performance due to mechanics like catastrophic forgetting. There is also other interesting behaviors were training on a broad training set can improve coding performance because there is positive transfer. There is 100% an effort to make solid coding focused models, but it is very hard to do that without including capabilities across a broad set of adjacent tasks. | ||
| ▲ | alansaber an hour ago | parent [-] | |
As you say- LLMs are fundamentally good because of their generalism. Distillation, ablation, ft all tend to be hacky and in some way hurt the model | ||