| ▲ | armcat 3 hours ago | |
I've been working in NLP for many years prior to the release of ChatGPT, and perplexity was generally associated with "human readability" rather than intelligence. So this is somewhat fascinating. There was an ACL paper from Tokyo University a couple of years ago that showed, across all languages, lower perplexity does not necessarily correlate with better "human likeness" [1]. I wonder if that's also the case here (where intelligence is concerned), that we are biasing too much towards English. [1] "Lower Perplexity is Not Always Human-Like", https://aclanthology.org/2021.acl-long.405v1.pdf | ||
| ▲ | dist-epoch 2 hours ago | parent | next [-] | |
They never say that their hold-out dataset to evaluate perplexity is just in English. Or written text for that matter (could also include code, ...) | ||
| ▲ | pishpash 2 hours ago | parent | prev [-] | |
When your performance is still worse than human of course any improvement makes it more human readable. But if you are in the regime beyond that... it'll head toward being more "oracle-like" and less "human-like" almost by definition. | ||