Remix.run Logo
thefourthchime 3 hours ago

Came here to say the exact same thing! People have to stop paying any attention to coding benchmarks that aren't Terminal Bench 4.

I noticed I noticed they didn't include Gemini 3.8, which also murders DeepSWE and Terminal Bench 2.0 -- because they are useless benchmarks now!

Of course in a couple months TB4 will also be old hat, so TB5 will have to be the new real benchmark.

dudeinhawaii an hour ago | parent [-]

Your post made me wonder if Artificial Analysis had finally moved to TB4 and lo and behold they have and Astra is tied with Fable 5.1 at 53.

That then made me realize that they lower the bars of tied scores so on the site it looks like Astra in second place. Weird. Anyway, yes, so many of these composite benchmark sites are irrelevant if they're not trimming the fat and sticking to the most up-to-date variants.