A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted?

What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?

▲

re5i5tor 4 hours ago | parent | next [-]

Have you tried it? I’d recommend doing so, it’s impressive in real use cases.

▲

aqme28 4 hours ago | parent | prev | next [-]

Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.

▲

deaux 3 hours ago | parent | next [-]

Here, filtered down for you. [0] Look at the individual benchmarks, not the combined one. You can tell that this model is much more benchmaxxed as its relative ranking swings between benchmarks is much larger. This is a hallmark.

[0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...

	▲	throwa356262 3 hours ago \| parent [-]
		I dont think this is benchmaxing. They have simply decided to not train the model in some areas such as world physics

▲

achrono 3 hours ago | parent | prev [-]

Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them.

I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.

	▲	skohan 3 hours ago \| parent \| next [-]
		At that point aren't you just edge-case testing? Surely most of your use-cases are not novel tasks that combine obscure domains. It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
	▲	CamperBob2 12 minutes ago \| parent \| prev [-]
		That problem sounds reminiscent of one I like to use as a benchmark, which is to request that the model create an .SVG of a logarithmic spiral of 50 numbered stones. Qwen 3.8 27B absolutely knocked that one out of the park, where a lot of larger models have failed outright or otherwise performed suboptimally. Can you share an example of the Greek-history puzzle prompts you're talking about?

▲

drob518 3 hours ago | parent | prev | next [-]

IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time. https://www.intelligence.ai/ http://arena.ai

▲

deaux 4 hours ago | parent | prev | next [-]

This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.

	▲	culi 3 hours ago \| parent [-]
		This website aggregates 82 different benchmarks. It seems to do pretty consistently well across the ones its been tested on https://benchlm.ai/models/qwen3-5-27b#ledger

▲

Iolaum 4 hours ago | parent | prev [-]

Yea and we are reaching the point where this benchmaxing is visible in the model's reported overthinking.

▲

logicchains 4 hours ago | parent [-]

It's not overthinking, it's the right amount of thinking necessary for such a small model to get good results. The dumber the model, the more it has to think to be smart. There's no easy way to reduce the thinking without reducing the model quality.

▲

zdragnar 4 hours ago | parent [-]

Qwen doom loops were amusing to watch the first time or two, but it's incredibly vexing to have it waffle over the same decision over and over and over and over again. I can get more done with a faster model by correcting it, and it feels better to babysit them than it does to babysit qwen to see if I need to intervene or if it will actually finish.

I do like the output from qwen when I get it, but honestly I haven't been impressed enough with it to put up with the downsides.

	▲	skohan 3 hours ago \| parent [-]
		It's only been a couple days, but I haven't seen looping issues with 3.8 so far, compared to 3.6 which did occasionally have this problem.