I started thinking about this back after the Llama 4 release, and since then our team has put a lot of thought into designing evaluations that don't saturate, are resistant to contamination, and can scale. What has worked best for us is using multi-agent environments with open-ended cooperative or competitive goals. Mostly designed as multiplayer games. The results tend to align with our experience for coding better than any non-aggregator benchmark, and likely at lower cost to run.

Data at https://gertlabs.com/rankings

▲

erikwiffin 3 hours ago | parent | next [-]

I've developed a benchmark that I think should be resistant to saturation, is easily verifiable, and anecdotally correlates with desirable behavior (ability to not get confused while generating text with state).

I think it's interesting, I think other people would find it useful, but I don't want to spend a bunch of money running it against all the frontier models.

What's the best way to reach out to labs like yours to collaborate on something like that? Are there any labs that are more open to submissions from internet randos?

▲

nwienert 4 hours ago | parent | prev [-]

If you're ranking Opus > Fable you're ranking "do [clearly defined thing with easy to grade endpoint]" too much. Real world doesn't value that nearly as much and it's why benchmarks are maxxed.

	▲	gertlabs 3 hours ago \| parent [-]
		That's a different problem than benchmark saturation, and it's something that we are actively working on measuring objectively. I agree that Opus 5 is not a great model, despite being clearly intelligent. It seems like a personality problem in user-driven agentic coding workflows, not a real capability issue. Not incorporating unspoken user intent, going off topic, incorporating some of the pedantry you find in GPT 5.x models, etc. That's also likely why Opus 5 ranks low on our "Social Intelligence" benchmark (https://gertlabs.com/rankings?mode=decision), although sample sizes on this one are still low.