well no of course not ... but they do this for qwen 3.8 27b, and decision models like this https://artificialanalysis.ai/articles/artificial-analysis-d...

this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.

Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.

I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest

perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.

And you go over to HF and it's even more mysterious https://huggingface.co/models?pipeline_tag=text-classificati...

Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...

Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.

This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.

So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.

I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.