Remix.run Logo
▲ underdeserver an hour ago

I wonder why they didn't show Gemini 4 Carbon's scores on the science benchmarks, especially since Google seems to be pushing it for that -

https://x.com/pushmeet/status/2120814064863461440?s=20

(FD: Google employee, I don't work on Gemini)

▲tedsanders an hour ago | parent | next [-]

I work at OpenAI and I made the plots. Gemini 4 Carbon was omitted when we couldn’t find public data, usually cost. No intent to hide anything.

If we missed anything, feel free to point me to it and we can add it.

▲underdeserver 43 minutes ago | parent [-]

So all the plots are based off public data? You don't run the models internally yourselves?

(Again speaking only for myself)

▲holzhaker1 an hour ago | parent | prev | next [-]

they are probably trying to authenticate their phone number to access agy

▲knuppar an hour ago | parent | prev | next [-]

Yeah it feels shady how Carbon was selectively plotted (or not), unlike Opus/Fable

▲eknkc an hour ago | parent | prev | next [-]

Probably still waiting for it to finish reasoning on the benchmark. Should be ready in a week or two.

▲TiredOfLife an hour ago | parent | prev [-]

Two posibilities. 1. They couldn't figure out how to actually use those Google models.

2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.

▲amluto 18 minutes ago | parent [-]

It's been most of year since I tried to use Google models, but I spent a couple of hours, without success, trying to pay for Gemini in a manner that didn't result in it training on my data. I did try asking Gemini, and it had no idea. Of course, I did not try using Gemini in an agent harness to pay for itself, partially because there was a bit of a chicken and egg problem [0]. :)

Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.

Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)

[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.