> The more plausible story is SVGmaxxing

Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.

▲

beering 12 hours ago | parent [-]

Really awful how the AI labs are skillmaxxing /s

Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.

	▲	dbt00 11 hours ago \| parent \| next [-]
		It's a problem because of Goodhart's law. If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.
	▲	Dylan16807 6 hours ago \| parent \| prev \| next [-]
		Avoiding "benchmaxxing" helps keep benchmarks be a good quantitative way to compare models! It's hard to come up with new benchmarks, and it's a waste of everyone's time if we have to churn through them constantly. Effort to keep AI labs working on general skills and not "teaching to the test" is effort well spent.
	▲	Wowfunhappy 12 hours ago \| parent \| prev [-]
		> If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! I don't think I'd go that far! When someone says a model has been benchmaxxed, what they really mean is that it performs better in benchmarks compared to their real world experience. That's a real thing, I've certainly experienced it with some models. ...my take is that some things in life just resist quantitative measurements. Who is the best job candidate? What is the best programming language? Add AI models to the pile.