| ▲ | nico 2 hours ago | |
> Detachment from actionable evaluations. Perplexity does not directly correspond to a task we care about The post makes a good case that at frontier scale this doesn't matter much, because perplexity gains turn into new capabilities. However, this has been an issue for me when training local models Some agents wanted to optimize for perplexity, and we did all kinds of experiments, but at least for what I wanted (a small conversational chat model that can run in my browser, ~8M params), that was the wrong way to go. I could get excellent ppl numbers, yet still almost no coherence What worked best, in my case, was to get better data, specifically, hand-crafted examples for the tasks I wanted the model to be good at. I'm guessing that perplexity might track capability when the model is big enough to turn the gains into general skills, but at a few million params it mostly buys you better-sounding word statistics | ||