| ▲ | Retro_Dev 3 hours ago | ||||||||||||||||||||||
Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is. | |||||||||||||||||||||||
| ▲ | jrflo 3 hours ago | parent | next [-] | ||||||||||||||||||||||
The dataset matters a lot here. You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph. If you measured the perplexity on any model on reciting the declaration of independence it would probably be zero. | |||||||||||||||||||||||
| |||||||||||||||||||||||
| ▲ | ThouYS 3 hours ago | parent | prev | next [-] | ||||||||||||||||||||||
depends entirely on the dataset... | |||||||||||||||||||||||
| ▲ | poszlem 26 minutes ago | parent | prev [-] | ||||||||||||||||||||||
log10(5,7) = 0,756 ln(5,7) = 1,74 did you even use the right logarithm? because they use the natural logarithm.. edit: that puts it before GPT-3.5 even | |||||||||||||||||||||||