how do you know it's benchmaxxed?

solenoid0937 5 hours ago | parent | next [-]

Friends at Meta with access to the model + personal experience at Meta.

Meta's performance process is essentially "show good numbers or you're out." So guess what people do when they don't have good numbers? They fudge them. Happens all across the company.

▲

luma 4 hours ago | parent | prev | next [-]

For one, they aren't using the latest version of many of the benchmarks. eg, ARC-AGI 2 and not 3, etc.

▲

prodigycorp 5 hours ago | parent | prev [-]

meta's benchmaxing tendencies are well known. llama4 was mega benchmaxxed, there's nothing that suggests to me that meta's culture has changed.

	▲	spindump8930 3 hours ago \| parent [-]
		Re: changes, there's been enormous turnover in AI organizations, and in theory this one was developed by a "new" org. Whether that means less or more benchmaxxing is anyone's guess.