| ▲ | declanjackson 27 minutes ago | |||||||||||||||||||
Something that is interesting is how it's positioned in the overall fleet of Gemini models. Although overall more Intelligent than Carbon, Carbon is still positioned as the 'long horizon' model. This is clear on AA Index, where we see higher overall scores for Neon but lower scores in evals like CritPt which test research-scale reasoning capabilities. With the frontier getting more jagged, we might see more of this 'less intelligent overall but useful for the frontier tasks'-type models, meaning picking a model will only get more difficult from here. (FD: I work at AA) | ||||||||||||||||||||
| ▲ | abixb 24 minutes ago | parent | next [-] | |||||||||||||||||||
Yes, I feel the next differentiation in model capabilities would be wrt how 'long of a horizon' goal they could autonomously work toward without breaking. We're in need for a new generation of evaluations and benchmarks for this -- METR doesn't cut it anymore. | ||||||||||||||||||||
| ▲ | seizethecheese 19 minutes ago | parent | prev | next [-] | |||||||||||||||||||
That would clear up why Carbon is such an outlier in terms of output tokens per task. | ||||||||||||||||||||
| ▲ | mi_lk 6 minutes ago | parent | prev | next [-] | |||||||||||||||||||
what's AA | ||||||||||||||||||||
| ||||||||||||||||||||
| ▲ | WarmWash 8 minutes ago | parent | prev | next [-] | |||||||||||||||||||
Neon is probably distilled from Google's internal successor to Carbon. | ||||||||||||||||||||
| ▲ | drivebyhooting 20 minutes ago | parent | prev [-] | |||||||||||||||||||
What we need is a model that automatically delegates intelligently (via RL and benchmaxxing?) so I can be freed of this burden. | ||||||||||||||||||||
| ||||||||||||||||||||