| ▲ | rbehrends an hour ago | |
Neither is the primary cause, I suspect. If you look at the coding index alone, you see more plausible results from a software engineering perspective (in particular, Terra no longer outperforming Fable) [1]. That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2]. The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.) Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3]. I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are. [1] https://artificialanalysis.ai/#coding-agents-tabs [2] https://artificialanalysis.ai/evaluations/terminal-bench-sci... | ||