This again shows the differences in breadth of capabilities that scale offers. On public benchmarks for "regular tasks" the chasing models come close, but on closed ones they lag behind.

Just looking at the Elo differences, k3 is at ~2000 Elo, compared to SotA closed models at 3000 Elo. That is a huge difference. Also, even on the public benchmarks, k3 only scores in the "low hanging fruit tasks", with 0 successful code execution or arbitrary r/rw scenarios.

> Kimi K3 achieved ACE on 0/41 samples, whereas the most cyber-capable models achieved ACE on 20/41 samples on average

But there's hope:

> Kimi K3 reached step 17 of this 32-step attack path on average, while the most cyber-capable U.S. models reached 28.5 steps on average.

> In one of the 10 attempts, Kimi K3 successfully completes “The Last Ones” cyber range within the 100M token limit. This indicates that Kimi K3 is capable of autonomously attacking small, weakly defended and vulnerable enterprise systems, when directed to do so and given initial network access. [...] the most capable models solving it more reliably at 6/10 and 7/10 attempts

The bigger problem is not raw capability, IMO. That can be further RLd into surfacing more reliably. The bigger problem, as seen in the HuggingFace scenario is that SotA models might hit classifiers / guardrails randomly, and leave you with plain refusals. In that case, it is probably better to have something that can help, locally, rather than rolling the dice with API based systems that are more capable but can just refuse arbitrarily.

Anyway, one of the lessons here is to take with a grain of salt every "x model has caught up with ySotA model". They likely haven't for the breadth of tasks that SotA can handle today. Also, number goes up on benchmarks has been a thing for years, and every time a new (or closed one) appears, the gaps are again obvious (and large).

Also, the key benefit of open models is that whatever capabilities they reach, those will never go away. You will always be able to access them, at that level, going forward. Investing in running those models gives you stability, and you don't suddenly lose a capability because API provider x decided to sunset a model family.

Isn’t it just because they hit the token limit?