Unfortunately Gemini isn't the only culprit here. I've had major problems with ChatGPT reliability myself.

Small things like this or the fact that AI studio still has issues with simple scrolling confuse me. How does such a brilliant tool still lack such basic things?

The latest comment on that issue is someone saying there's a fix available for you to try.

chatgpt also has lots of reliability issues

I wonder if this is because a memory cap was reached at that output token. Perhaps they route conversations to different hardware depending on how long they expect it to be.

Yes agree, it was totally broken when I tested the API two months ago. Lots of failed to connect and very slow response time. Hoping the update fixes these issues.

That used to happen a lot in ChatGPT too.

FWIW, I think GLM-4.5 or Kimi K2 0905 fit the bill pretty well in terms of complete and consistent.

(Disclosure: I'm the founder of Synthetic.new, a company that runs open-source LLMs for monthly subscriptions.)

I wonder if [good examples of] SVGs of pelicans on bikes are "being introduced" into training sets. Some of the engineers who work on this stuff are the kind to hang out here.

Who wins in the end? the frogs? the ducks? or the pelicans?

That's why people called the second version of Sonnet v3.5 simply v3.6, and Anthropic acknowledged that by naming the next version v3.7

It's pretty common to refer to models by the month and year they were released.

For example, the latest Gemini 2.5 Flash is known as "google/gemini-2.5-flash-preview-09-2025" [1].

[1]: https://openrouter.ai/google/gemini-2.5-flash-preview-09-202...

2.5 is not the version number, it's the generation of the underlying model architecture. Think of it like the trim level on a Mazda 3 hatchback. Mazda already has the Mazda 3 Sport in their lineup, then later they release the Mazda 3 Turbo which is much faster. When they release this new version of the vehicle its not called the Mazda 4... that would be an entirely different vehicle based on a new platform and powertrain etc (if it existed). The new vehicle is just a new trim level / visual refresh of the existing Mazda 3.

That's why Google names it like this, but I agree its dumb. Semver would be easier.

Maybe they’re signalling it’s more of a bug fix?

I would be surprised if this dichotomy you're painting holds up to scrutiny.

My understanding is Gemini is not far behind on "intelligence", certainly not in a way that leaves obvious doubt over where they will be over the next iteration/model cycles, where I would expect them to at least continue closing the gap. I'd be curious if you have some benchmarks to share that suggest otherwise.

Meanwhile, afaik something Google has done, and perhaps relates back to your point re "latency/TPS/cost dimensions" that other providers aren't doing as much is integrating their model into interesting products beyond chat, at a pace that seems surprising given how much criticism they had been taking for being "slow" to react to the LLM trend.

Besides the Google Workspace surface and Google search, which now seem obvious - there are other interesting places where Gemini will surface - https://jules.google/ for one, to say nothing of their experiments/betas in the creative space - https://labs.google/flow/about

Another I noticed today: https://www.google.com/finance/beta

I would have thought putting Gemini on a finance dashboard like this would be inviting all sorts of regulatory (and other) scrutiny... and wouldn't be in keeping with a "slow" incumbent. But given the current climate, it seems Google is plowing ahead just as much as anyone else - with a lot more resources and surface to bring to bear. Imagine Gemini integration on Youtube. At this point it just seems like counting down the days...

IMO the race for Latency/TPS/cost is entirely between grok and gemini flash. No model can touch them (especially for image to text related tasks), openai/anthropic seem entirely uninterested in competing for this.

> because these Gemini models sometimes feel downright lobotomized compared to claude or gpt-5.

I'm using Gemini (2.5-pro) less and less these days. I used to be really impressived with its deep research capabilities and ability to cite sources reliably.

The last few weeks, it's increasingly argumentative and incapable of recognizing hallucinations around sourcing. I'm tired of arguing with it on basics like RFCs and sources it fabricates, won't validate, and refuses to budge on.

Example prompt I was arguing with it on last night:

> within a github actions workflow, is it possible to get access to the entire secrets map, or enumerate keys in this object?

As recent supply-chain attacks have shown, exfiltrating all the secrets from a Github workflow is as simple as `${{ toJSON(secrets) }}` or `echo ${{ toJSON(secrets) }} | base64` at worse. [1]

Give this prompt a shot! Gemini won't do anything except be obstinately ignorant. With me, it provided a test case workflow, and refused to believe the results. When challenged, expect it to cite unrelated community posts. Chatgpt had no problem with it.

[1] https://github.com/orgs/community/discussions/174045 https://github.com/orgs/community/discussions/47165

We had to drop Gemini api cause it was so unreliable in production, no matter how long you waited.

Can't agree with that. Gemini doesn't lead just on price/performance - ironically it's the best "normie" model most of the time, despite it's lack of popularity with them until very recent.

It's bad at agentic stuff, especially coding. Incomparably so compared to Claude and now GPT-5. But if it's just about asking it random stuff, and especially going on for very long in the same conversation - which non-tech users have a tendency to do - Gemini wins. It's still the best at long context, noticing things said long ago.

Earlier this week I was doing some debugging. For debugging especially I like to run sonnet/gpt5/2.5-pro in parallel with the same prompt/convo. Gemini was the only one that, 4 or so messages in, pointed out something very relevant in the middle of the logs in the very first message. GPT and Sonnet both failed to notice, leading them to give wrong sample code. I would've wasted more time if I hadn't used Gemini.

It's also still the best at a good number of low-resource languages. It doesn't glaze too much (Sonnet, ChatGPT) without being overly stubborn (raw GPT-5 API). It's by far the best at OCR and image recognition, which a lot of average users use quite a bit.

Google's ridiculously bad at marketing and AI UX, but they'll get there. They're already much more than just a "bang for the buck" player.

FWIW I use all 3 above mentioned on a daily basis for a wide variety of tasks, often side-by-side in parallel to compare performance.

The other day I heard gpt-5 was really an efficiency update

This is also my beef...

Anthropic kind of did the same thing [1] except it back-fired recently with the cries of "nerfing".

We buy these tokens, which are very hard to do in limited tiers, they expire after only a year, and we don't even know how often the responses are changing in the background. Even a 1% improvement or reduction I would want disclosed.

Really scary foundation AI companies are building on IMO. Transparency and access is important.

[1] https://status.claude.com/incidents/h26lykctfnsz

I wouldn't call that a nitpick, it's a major annoyance. Version numbers become useless with that kind of policy.

I would assume that it will supersede the model that they currently have. So eventually 2.5 flash will be the new and improved 2.5 Flash rather than 2.6.

Same way that openai updated their 4-o models and the like, which didn't turn out so well when it started glazing everyone and they had to revert it (maybe that was just chat and not api)

I think “Non-AI summary” is going to become a thing. I already enjoyed reading it more because I knew someone had thought about the content.

I'm stealing "Non-AI Summary"

Any idea what "output token efficiency" refers to? Gemini Flash is billed by number of input/output tokens, which I assume is fixed for the same output, so I'm struggling to understand how it could result in lower cost. Unless of course they have changed tokenization in the new version?

Thank you for this, seems like an iterative improvement.

2.5 Flash is the first time I've felt AI has become truly useful to me. I was #1 AI hater but now find myself going to the Gemini app instead of Google search. It's just better in every way and no ads. The info it provides is usually always right and it feels like I have the whole generalized and accurate knowledge of the internet at my fingertips in the app. It's more intimate, less distractions. Just me and the Gemini app alone talking about kale's ideal germination temperature, instead of a bunch of mommy bloggers, bots, and SEO spam.

Now how long can Google keep this going and cannibalizing how they make money is another question...

I've been finding it leaps and bounds above other models but I'm only using it via aistudio. I haven't tried any IDE integration or similar, so can't talk to that. I do still have to tell it to stop it with the effusive praise (I guess that also helps reduce context windows)

I agree. I think it comes down OpenAI's superior post-training.

ChatGPT is better at:

A) Interpreting what I'm asking it for me needing to provide additional explicit context.

B) Formatting answers in a way that are easily digestible.

Maybe you are using it wrong.

I wonder how Gemini subscribers feel!

Gemini 2.5 Flash runs circles around ChatGPT 5 for many of my tasks, I’m surprised it’s not more popular than it is.

Not sure prices are changed though. :/

My one big problem with OpenRouter is that, as far as I can tell, they don't provide any indication of how many companies are using each model.

For all I know there are a couple of enormous whales on there who, should they decide to switch from one model to another, will instantly impact those overall ratings.

I'd love to have a bit more transparency about volume so I can tell if that's what is happening or not.

I know we have a lot of workloads at my company on older models no one has bothered to upgrade yet

Price, 2.0 Flash is cheaper than 2.5 Flash but still very good model.

Maybe the same reason why they kept the name for the 2.5 Flash update.

People are lazy at pointing to the latest name.

Why is Grok so popular

Gemini 2.0 Flash is the best fast non reasoning model by quite a margin. Lot of things doesn't require any reasoning.

It’s cheaper and faster. What’s not to understand?

Did they? I'm looking at the Artificial Analysis leaderboard site now and I only see price as USD/1M tokens.

If you want stable models I think you could get that through Azure.

There several reasons responses from the same model might vary:

- "temperature" - intentional random sampling from the most likely next tokens to improve "creativity" and help avoid repetition

- quantization - running models with lower numeric precision (saves on both memory and compute, without impacting accuracy too much)

- differences in/existence of a system prompt, especially when using something end-user-oriented like Qwen Chat

- not-quite-deterministic GPU acceleration

Benchmarks are usually run at temperature zero (always take the most likely next token), with the full-precision weights, and no additions to the benchmark prompt except necessary formatting and stuff like end-of-turn tokens. They also usually are multiple-choice or otherwise expect very short responses, which leaves less room for run-to-run variance.

Of course a benchmark still can't tell you everything - real-world performance can be very different.

That's a difference in the system prompt, not the model itself.

I can't speak to qwen, but something interesting with Deepseek is that the official API supports almost no parameters, while the vllm hosts on openrouter do. The experience you get with the rehosters is wildly different since you can use samplers.

Yeah, why is it that working with AI makes people completely forget what version numbers mean?

gemini-2.5-flash-preview-09-2025 - what are they thinking?

I thought about joking that they had AI name it for them, but when I asked Gemini, it said that this name was confusing, redundant, and leads to unnecessarily high cognitive load.

Maybe Googlers should learn from their own models.

Because the number is model generation.

I think that the Gemini 3 pro might be next month I am not sure.

can I get the sources of your rumour please? (Yes I know that I can search it but I would honestly prefer it if you could share it, thanks in advance!)

ChatGPT and Claude are mistaken if they think it is incorrect. The parallelism in verb tenses is between "continuing to deliver" and "improving the efficiency". It's a bit wordy, but definitely not wrong.

"Improving the efficiency" sounds fine to me (a native English speaker), what's wrong with it in your opinion?

This is pedantic. It's perfectly fine usage in non-formal English speaking. What's more - who gives a shit? By your own standards, you're inserting a quote in the middle of your comment in an arguably similarly "awkward" way.

This is not my experience. In my experience Gemini 2.5 Pro is the best model in every use-case I tried. There are a few very hard (graduate level) logic or math problems that Claude 4.1 Opus edged-out over Gemini 2.5 Pro, but in general if you have no idea which model will perform best on a difficult question, imho Gemini 2.5 Pro is a safer bet especially since it's significantly cheaper. Gemini 2.5 Flash is really good but imho not nearly as good as Pro in (1) research math (2) creative/artistic writing (3) open ended programming debugging.

On the other hand, I do prefer using Claude 4 Sonnet on very open-ended agentic programming tasks because it seems to have a better integration with VSCode Copilot. Gemini 2.5 Pro bugs out much more often where Claude works fine almost every time.

Yeah that's how I feel too. Flash is less verbose and every LLM nowadays seems to be designed by some low-taste people who reward the model for falsely hedging (i.e. "The 2024 Corolla Cross usually has an X gallon gas tank") on stuff that isn't at all variable or questionable. This false hedging is way more of an issue than hallucinations in my experience and the "smarter" 2.5 Pro is not any better at avoiding this issue than Flash

Also 2.5 Pro is often incapable of searching and will hallucinate instead. I don't know why. It will claim it searched and then return some made up results instead. 2.5 Flash is much more consistently capable of searching

I tried to put Pro deep research on an actual research task and it didn’t even return anything just kept on working.

2.5 isn't the version number, its the model generation. it would only be updated when the underlying model architecture, training, etc are updated. this release is, as the name implies, the same model but likely with hardware optimizations, system prompt, and fine-tuning tweaks applied.

FWIW, the versions are not semver but they do follow a defined and regular version schema: https://ai.google.dev/gemini-api/docs/models#model-versions.

The version number is about the architecture of the model, the date is just about the last weights of the model.

This is a performance update to a previous generation model. It's not a new model.

And this existing model’s update is evidence how? What were your expectations of this update?

I actually even agree that the progress is plateauing, but your comment is a non-sequitur.

Not really. A lot of new amazing Qwen models just dropped.