Remix.run Logo
▲ jiehong 3 hours ago

Isn’t this a problem? How can we compare and hold companies accountable if they measure this perplexity on different datasets?

Should we trust that the dataset of each company is big and diverse enough to be representative? I don’t hope too much…

▲programjames 2 hours ago | parent [-]

Presumably they could each create a graph for their own models and internal datasets, then use public benchmarks to calibrate between them.