I agree, however It's too easy to game arbitrary limits like "medium". There is no olympics for "just rolled out of bed and slightly hungover"

▲

leidenfrost a day ago | parent | next [-]

We don't even need a standard for reasoning levels.

Just accept whatever the provider names as "medium" and compare the accuracy between models vs the time it takes to complete a task.

	▲	SamuelAdams 12 hours ago \| parent [-]
		The fashion industry has been doing this for decades. Medium means different things for different brands.

▲

RussianCow a day ago | parent | prev | next [-]

You don't need to directly compare the same reasoning levels. Today you can compare Sol on Low vs Haiku on Max, for example, but you can't look at GLM 5.3 High, only Max.

▲

jchw a day ago | parent | prev | next [-]

For some tasks I genuinely feel Opus 5.5 does better on low than medium or high. The trouble, as always, is figuring out which.

Edit: I realize after commenting that you're not suggesting medium is useless, but just arbitrary and differs between models. Personally I don't want to directly compare mediums between models, I just want to be able to compare across effort levels.

▲

disiplus a day ago | parent [-]

i run opus on medium, sonet on high, and haiku on high for implementation based how complicated is for planing and specing and review opus is on high usually, except if if the task is contained.

did not try to run it on low, will have to try it.

▲

jchw a day ago | parent [-]

Opus 5.5 on low is good if you are relatively sure a given task is straightforward and time spent gathering background information is purely wasted or possibly even detrimental. Opus 5.5 low seems great at doing the most obvious thing based on the prompt without thinking much about it.

	▲	nl a day ago \| parent [-]
		I agree - Opus 5.5 on low is a really great model if it has some direction and the tasks aren't complex. If the tasks are complex cranking Opus up to high or xhigh and getting it to delegate to Opus low subagents is pretty effective and much faster than running everything at high.

▲

eli a day ago | parent | prev | next [-]

A graph of time (or cost) per task against success rate does it. Just include all the different levels.

	▲	Kinrany a day ago \| parent \| next [-]
		This exactly. The three interesting parameters are total time to answer, total cost, and output quality (for any given purpose).
	▲	arxo a day ago \| parent \| prev [-]
		[flagged]

▲

esafak a day ago | parent | prev [-]

That's the beauty of the line charts; the labels are arbitrary points on the line. You can easily compare the series visually.