Great benchmark!

I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.

Is there a reasonable way to run a representative subset myself?

Edit:

> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.

And https://github.com/gabeorlanski/scb-problems/

For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.

I wish a company named human layer won't write their reports with so much Claudisms.

The irony is the article is very sloppy. Can we at least write it and review it properly..