Speaking of arena comparisons of K3.1 and DS4.1-Flash, we're spinning one up that's more relevant than design, I think: https://agentduel.com/leaderboard

Here, DS4.1-Flash and K3.1 are basically tied, with K3.1 being slower and more expensive. Big error bars, though.

Even with error bars, DS4.1-flash is clearly an impressive model.

▲

reissbaker 6 hours ago | parent [-]

If your benchmark puts DS4.1-Flash at the same tier as K3 or K3.1, I'm sorry to say your benchmark is broken. It really isn't, as much as I wish it was.

▲

seizethecheese 6 hours ago | parent [-]

We allow early voting. This is against what most arenas do, but is on purpose. If the model is faster to do simple stuff, it should be chosen. In this sense, DS4.1-Flash is better on many tasks.

You can remove early voting from leaderboard and they are no longer the same.

	▲	purerandomness 21 minutes ago \| parent \| next [-]
		At the point where I choose an expensive model like Kimi 3.1, speed doesn't matter, at all. Usually, it's because all else failed, and I have to resort to burning Tokens with more expensive/better models, so a few minutes here and there really should not matter.
	▲	nostrebored an hour ago \| parent \| prev [-]
		But how do you verify a fair or representative distribution of complexity?