Remix.run Logo
fastball 6 hours ago

Yes, this (imo) is a clear result of benchmaxxing. You can get a much better score on most "intelligence" benchmarks by massively over-saturating reasoning. This looks good on those, but for actual daily usage makes the models much less effective: I don't want a model I use for coding to burn a bunch of reasoning (read: time) on trivial tasks.

bensyverson 5 hours ago | parent | next [-]

It's undeniable that some of these models generate a ton of thinking tokens, but it's arguable whether that makes them "much less effective."

For example, Kimi 2.7 has been really effective for me despite having verbose thinking blocks, simply because it runs so fast. Speed-wise, it feels about like Sonnet, possibly faster.

gpm 5 hours ago | parent | prev [-]

I strongly suspect the flip side is that in the future it enables you to train smarter models by "distilling" the end result of the super duper heavily thinking models.

esafak 5 hours ago | parent [-]

But these models already distill the smarter American ones ;)

lkasjdhf 2 hours ago | parent [-]

It's turtles all the way down