| ▲ | jrflo 3 hours ago | |||||||
The dataset matters a lot here. You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph. If you measured the perplexity on any model on reciting the declaration of independence it would probably be zero. | ||||||||
| ▲ | blueblisters 2 hours ago | parent | next [-] | |||||||
> You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph Ideally the hold out set is randomly sampled. You need a hold out set of Inklings pretraining run. | ||||||||
| ▲ | jiehong 2 hours ago | parent | prev [-] | |||||||
Isn’t this a problem? How can we compare and hold companies accountable if they measure this perplexity on different datasets? Should we trust that the dataset of each company is big and diverse enough to be representative? I don’t hope too much… | ||||||||
| ||||||||