new | show | ask | jobs Github

ricardobeat 19 hours ago

The page is currently tracking Opus 5, which was released July 24 and has not seen any updates since.

30-day average 83% [71-91], last result is 79% [66-88], and it dipped to 75% [61-85] a week ago.

▲

Aurornis 19 hours ago | parent [-]

The page says

> We always use the latest available Claude Code release and the SOTA model

They've stepped up the benchmark for each new model. Presumably they're gathering Fable data now.

They also link to Anthropic's public blog post about some degradations, their cause, and how they fixed them. The time period sounds like the "recent weeks" you experienced: https://www.anthropic.com/engineering/a-postmortem-of-three-...

▲

ricardobeat 19 hours ago | parent [-]

Fable 5.1 was released 20 days ago. That's a lot of time to gather data. Since they're still tracking Opus 5 anyway, there is no obvious reason the perf delta section would be disabled. Are you involved with the project?

The Anthropic post points to the latest fix on Sept 12, and the issues mentioned also only affected Sonnet 4 and Haiku. Opus was misbehaving just last week. You are choosing to not see the evidence of degration, 85% -> 75% is a generational dip in intelligence.

	▲	Aurornis 19 hours ago \| parent [-]
		Not involved, no. Just guessing. It doesn't look frequently updated. A lot of these daily-benchmarking sites popped up earlier this year. Most of them have faded away after they all failed to produce the smoking gun that everyone expected. This site survived because it kind of caught a dip one day, maybe. The results are really rather flat and daily benchmark runs are expensive, so most of these projects give up after a while.