Remix.run Logo
▲ riemannzeta 2 hours ago

Log-perplexity is cross-entropy, and its reducible part is the KL divergence between model and corpus. I've argued that the same quantity prices the gap between any two agents, human or AI: https://www.symmetrybroken.com/maintaining-divergence/

The floor Pachocki can't locate is fundamental. No model can predict the next word better than its underlying agreement with the author's model.

▲izenja an hour ago | parent | next [-]

For one author's model, sure. For explaining all the authors/text in the corpus, it seems like it would be best to understand the common parts and then the smallest explanation for how they diverge. For example, a teacher understands the material, and can spot the point where each student's solution went wrong, sending them toward a wrong answer.

▲riemannzeta an hour ago | parent [-]

That's true. The essay I linked was written with a focus on closing the gap between human and human or human and AI.

▲pishpash 2 hours ago | parent | prev [-]

Why would you want one model to predict another? They should all be measured against ground truth.

▲riemannzeta 2 hours ago | parent [-]

[flagged]