Remix.run Logo
▲ tristanj 4 hours ago

It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:

1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.

2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.

3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.

▲aberoham an hour ago | parent | next [-]

Google's frontend web servers routinely time out connections from agy (the antigravity cli) when using Carbon, wasting tons of work. Like any effort level more than "low" is completely unusable.

It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.

One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.

Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.

▲p_l an hour ago | parent | next [-]

This has been acknowledged by recent "agy" release notes, it was timing out in client not in backend

▲thanh81646 24 minutes ago | parent | prev [-]

[flagged]

▲fsh 3 hours ago | parent | prev | next [-]

4. Google did a worse job than Anthropic and OpenAI.

▲dpe82 3 hours ago | parent [-]

Well sure, but parent is trying to reason about why.

▲andy_ppp 2 hours ago | parent [-]

It’s unlikely to be deliberate tbh, it’s more likely the data, algorithms and the training environments aren’t quite as good as Anthropic, Open AI is a lot closer to Google than people want to admit.

▲zarzavat 3 hours ago | parent | prev | next [-]

It's 1. Carbon was announced the day of Google's earnings call. It's a PR stunt.

▲teknologist 3 hours ago | parent | prev | next [-]

Also, Google uses their own TPU architecture rather than GPU/Nvidia.

▲vintermann an hour ago | parent | prev [-]

So in short, as is usually claimed about Google products from bazel to tensorflow to Kubernetes: they may work better inside Google than outside of it.