| ▲ | riemannzeta 2 hours ago | |||||||
Log-perplexity is cross-entropy, and its reducible part is the KL divergence between model and corpus. I've argued that the same quantity prices the gap between any two agents, human or AI: https://www.symmetrybroken.com/maintaining-divergence/ The floor Pachocki can't locate is fundamental. No model can predict the next word better than its underlying agreement with the author's model. | ||||||||
| ▲ | izenja an hour ago | parent | next [-] | |||||||
For one author's model, sure. For explaining all the authors/text in the corpus, it seems like it would be best to understand the common parts and then the smallest explanation for how they diverge. For example, a teacher understands the material, and can spot the point where each student's solution went wrong, sending them toward a wrong answer. | ||||||||
| ||||||||
| ▲ | pishpash 2 hours ago | parent | prev [-] | |||||||
Why would you want one model to predict another? They should all be measured against ground truth. | ||||||||
| ||||||||