> All flagship models are within like 1-5% of each other

Don't know about that.

I'm using code review of my lone lisp project as a benchmark. It's a massive parallel code review where a coordinator cuts up the codebase into sections and dispatches agents to consider each part from different perspectives like quality, maintainability, consistency, correctness, rigor, etc.

Ran a complete Fable/max code review. Took over a month on a subscription. Now I've switched to OpenAI and am repeating the exact same review with Sol/max.

It's still not done yet but preliminary findings suggest Sol can only reproduce 70-90% of Fable's findings. So I think these models aren't as close as we've been led to believe.

[dead]