Remix.run Logo
▲ declanjackson 27 minutes ago

Something that is interesting is how it's positioned in the overall fleet of Gemini models. Although overall more Intelligent than Carbon, Carbon is still positioned as the 'long horizon' model. This is clear on AA Index, where we see higher overall scores for Neon but lower scores in evals like CritPt which test research-scale reasoning capabilities.

With the frontier getting more jagged, we might see more of this 'less intelligent overall but useful for the frontier tasks'-type models, meaning picking a model will only get more difficult from here.

(FD: I work at AA)

▲abixb 24 minutes ago | parent | next [-]

Yes, I feel the next differentiation in model capabilities would be wrt how 'long of a horizon' goal they could autonomously work toward without breaking. We're in need for a new generation of evaluations and benchmarks for this -- METR doesn't cut it anymore.

▲seizethecheese 19 minutes ago | parent | prev | next [-]

That would clear up why Carbon is such an outlier in terms of output tokens per task.

▲mi_lk 6 minutes ago | parent | prev | next [-]

what's AA

▲mohamedsayhii 4 minutes ago | parent [-]

Artificial Analysis

▲WarmWash 8 minutes ago | parent | prev | next [-]

Neon is probably distilled from Google's internal successor to Carbon.

▲drivebyhooting 20 minutes ago | parent | prev [-]

What we need is a model that automatically delegates intelligently (via RL and benchmaxxing?) so I can be freed of this burden.

▲iamdelirium 16 minutes ago | parent | next [-]

Antigravity has a Orchestrator model that does this.

▲refulgentis 10 minutes ago | parent | prev | next [-]

There's stuff that does this and I've put a substantial amount of time last couple months looking into it, unfortunately you always, always, end up at chicken-and-egg problems.

It's really frustrating to describe. Easiest is you can't score on just the prompt, you need the full context. Ex. "Solve Fermat's last theorem" initial prompt then "continue" followup prompt. Which gets you back in the tarpit of "if I have to pay a model to do an inference run on context, then, the context is cached and I should continue using that model."

The vision of another model thats RL maxx'd on knowing how to delegate has a few problems. One, we can observe externally, it died/didn't work even at OpenAI's scale and willingness to make it work, i.e. the...GPT-5.0? IIRC...router that silently went away. The other, if such a model was built, it'd also be good at regulating it's own thinking enough that it'd naturally use less tokens.

In practice, the frontier hackers have settled on a decision model (i.e. Jev) x calibrating effort level or choosing from a set of models x reasoning effort tuples they specify rules for choosing in their Jev prompt. My guess is this stays stable and the only real choice.

▲ihsw 8 minutes ago | parent | prev [-]

[dead]