I've been doing some research lately on my own (aside from the major benchmarking going on) and I've definitely found cases where medium models do much poorly and cost more than the higher models. But on the other hand, if I can give the medium models certain skills or other resources to complete a task then they perform better in cost than the higher model (which, given the same resources, sometimes gets more confused than the medium does).

At least for my use case of evaluations of a set of features.

▲

XCSme a day ago | parent [-]

Yup, in my benchmarks usually medium or high do better than xhigh or max

▲

spider-mario a day ago | parent [-]

You’re talking about something else (reasoning levels). The parent comment is referring to e.g. GPT Terra as opposed to Luna, Sol or Astra, Claude Sonnet as opposed to Haiku, Opus or Fable, or Gemini Flash/Hydrogen as opposed to Flash-Lite, Argon or Carbon.

	▲	XCSme a day ago \| parent [-]
		Oh, good catch. I do use Sol in production, for cases where I need quite good results, but can't afford to run in production the highest tier model due to pricing (so it's like the highest tier model I can afford for that use-case). To be more specific, this is for a support automation, where the model has to be quite accurate in following rules and generate responses, and Luna models or lower-tier models often make bad mistakes and generate wrong responses, so are not really usable for sending data directly to the user.