How can a benchmark be secret if you post it to an API to test a model on it?

"We totally promise that when we run your benchmark against our API we won't take the data from it and use to be better at your benchmark next time"

If you want to do it properly you have to avoid any 3rd party hosted model when you test your benchmark, which means you can't have GPT5, claude, etc. on it; and none of the benchmarks want to be 'that guy' who doesn't have all the best models on it.

So no.

They're not secret.

▲

dmos62 3 days ago | parent [-]

How do you propose that would work? A pipeline that goes through query-response pairs to deduce response quality and then uses the low-quality responses for further training? Wouldn't you need a model that's already smart enough to tell that previous model's responses weren't smart enough? Sounds like a chicken and egg problem.

▲

irthomasthomas 3 days ago | parent [-]

It just means that once you send your test questions to a model API, that company now has your test. So 'private' benchmarks take it on faith that the companies won't look at those requests and tune their models or prompts to beat them.

▲

dmos62 3 days ago | parent | next [-]

Sounds a bit presumptious to me. Sure, they have your needle, but they also need a cost-efficient way to find it in their hay stack.

	▲	lucianbr 3 days ago \| parent \| next [-]
		They have quite large amounts of money. I don't think they need to be very cost-efficient. And they also have very smart people, so likely they can figure out a somewhat cost-efficient way. The stakes are high, for them.
	▲	noodletheworld 3 days ago \| parent \| prev [-]
		Security through obscurity is not security. Your api key is linked to your credit card, which is linked to your identity. …but hey, youre right. Lets just trust them not to be cheating. Cool.

▲

merelysounds 3 days ago | parent | prev [-]

Would the model owners be able to identify the benchmarking session among many other similar requests?

	▲	irthomasthomas 3 days ago \| parent [-]
		Depends. Something like arc-agi might be easy as it follows a defined format. I would also guess that the usage pattern for someone running a benchmark will be quite distinct from that of a normal user, unless they take specific measures to try to blend in.