Since they did this with their own harness I m not sure it's apples to apples comparison.

NitpickLawyer 12 hours ago | parent | next [-]

> not sure it's apples to apples comparison.

They're literally comparing the previous version of the same model with the new one. It's based on the same architecture, same pre-trained model, just different post-training. It doesn't get more apples to apples than this.

▲

dnhkng 12 hours ago | parent [-]

I think the commenter means the Flash vs Terra benchmarks.

	▲	NitpickLawyer 10 hours ago \| parent [-]
		Ah, my bad. Yeah that makes sense. They do say "The official V4-Flash natively supports the Responses API format and is specifically adapted for Codex.", so at some point someone will make a "same harness" comparison.

▲

dnhkng 12 hours ago | parent | prev [-]

It will be fair if they release the harness though. I think now the future will be paired model-harness releases, not just weight dumps.

The performance changes are so big with the right harness that is makes sense to engineer the harness and fine-tune the model to one another from the start.