Gemini 3 Pro Preview gets 96.8% on the same benchmark? That's impressive

the only benchmark i trust

Is that the first SVG pelican with drop shadows?

What happens if you ask for a pterodactyl on a motorbike?

Would like to know how much they are optimizing for your pelican....

seems to be eating something

> For coding tasks, GPT-5.1-Codex-Max is a faster, more capable, and more token-efficient coding variant

Hm, yeah, strange. You would not be able to tell, looking at every chart on the page. Obviously not a gotcha, they put it on the page themselves after all, but how does that make sense with those benchmarks?

gemini does, grok does, nobody else does (except alibaba but it’s not there yet)

I have found Claude‘s voice chat to be better. I only recently tried it because I liked ChatGPTs enough, but I think I’m going to use Claude going forward. I find myself getting interrupted by ChatGPT a lot whenever I do use it.

gemini live is a thing - never tried chaptgpt, are they not similar?

I can't keep up with half the new features all the model companies keep rolling out. I wish they would solve that

Are you saying ChatGPT's voice chat is of good quality? Because for me it's one of its most frustrating weaknesses. I vastly prefer voice input to typing, and would love it if the voice chat mode actually worked well.

But apart from the voices being pretty meh, it's also really bad at detecting and filtering out noise, taking vehicle sounds as breaks to start talking in (even if I'm talking much louder at the same time) or as some random YouTube subtitles (car motor = "Thanks for watching, subscribe!").

The speech-to-text is really unreliable (the single-chat Dictate feature gets about 98% of my words correct, this Voice mode is closer to 75%), and they clearly use an inferior model for the AI backend for this too: with the same question asked in this back-and-forth Voice mode and a normal text chat, the answer quality difference is quite stark: the Voice mode answer is most often close to useless. It seems like they've overoptimized it for speed at the cost of quality, to the extent that it feels like it's a year behind in answer reliability and usefulness.

To your question about competitors, I've recently noticed that Grok seems to be much better at both the speech-to-text part and the noise handling, and the voices are less uncanny-valley sounding too. I'd say they also don't have that stark a difference between text answers and voice mode answers, and that would be true but unfortunately mainly because its text answers are also not great with hallucinations or following instructions.

So Grok has the voice part figured out, ChatGPT has the backend AI reliability figured out, but neither provide a real usable voice mode right now.

I'm a big user of Gemini voice. My sense is that Gemini voice uses very tight system prompts that are designed to give you an answer and kind of get you off the phone as much as possible. It doesn't have large context at all.

That's how I judge quality at least. The quality of the actual voice is roughly the same as ChatGPT, but I notice Gemini will try to match your pitch and tone and way of speaking.

Edit: But it looks like Gemini Voice has been replaced with voice transcription in the mobile app? That was sudden.

Their voice agent is handy. Currently trying to build around it.

try gemini voice chat

Try elevenlabs

Qwen does.

Flagship models have rarely being cheaper, and especially not on release day. Only a few cases of this really.

Notable exceptions are Deepseek 3.2 and Opus 4.5 and GPT 3.5 Turbo.

The price drops usually are the form of flash and mini models being really cheap and fast. Like when we got o4 mini or 2.0 flash which was a particularly significant one.

Gemini 3 Pro Preview also got more expensive than 2.5 Pro.

2.5 Pro: $1.25 input, $10 output (million tokens)

3 Pro Preview: $2 input, $12 output (million tokens)

Reading this comment, it just occurred to me that we're still in the first phase of the enshittification process.

Previous model's prices usually go down, but their flagship has always been the most expensive one.

It also seems much more "smarter" though

We're also in benchmark saturation territory. I heard it speculated that Anthropic emphasizes benchmarks less in their publications because internally they don't care about them nearly as much as making a model that works well on the day-to-day

Note that GPT 5.2 newly supports a "xhigh" reasoning level, which could explain the better benchmarks.

It'll be noteworthy to see the cost-per-task on ARC AGI v2.

That ARC AGI score is a little suspicious. That's a really tough for AI benchmark. Curious if there were improvements to the test harness because that's a wild jump in general problem solving ability for an incremental update.

I don't think SWE Verified is an ideal benchmark, as the solutions are in the training dataset.

For a minor version update (5.1 -> 5.2) that's a way bigger improvement than I would have guessed.

Yes, but it's not good enough. They needed to surpass Opus 4.5.

Open AI has already been busted for getting benchmark information and training the models on that. At this point if you believe Sam Altman, I have a bridge to sell you.

Where did you get that from? Cutoff date says august 2025. Looks like a newly pretrained model

Apparently they have not had a successful pre training run in 1.5 years

I thought whenever the knowledge cutoff increased that meant they’d trained a new model, I guess that’s completely wrong?

The irony is that Deepseek is still running with a distilled 4o model.

I’m extremely skeptical because of all those articles claiming OpenAI was freaking out about Gemini - now it turns out they just casually had a better model ready to go? I don’t buy it.

Unfortunately there are never any real specifics about how any of their models were trained. It's OpenAI we're talking about after all.

100% on the AIME (assuming its not in the training data) is pretty impressive. I got like 4/15 when I was in HS...

to be fair I've seen the other sota models do this as well

I'm not sure why being a MoE model would allow OpenAI to "turn up the good stuff". You can't just increase the number of E without training it as such.

GPT 4o was an MoE model as well.

#1 models are usually priced at 2x more than the competition, and they often decrease the price right when they lose the crown.

Due to the "Code Red" threat from Gemini 3, I suspect they'll hold off throttling for longer than usual (by incinerating even more investor capital than usual).

Jump in and soak up that extra-discounted compute while the getting is good, kids! Personally, I recently retired so I just occasionally mess around with LLMs for casual hobby projects, so I've only ever used the free tier of all the providers. Having lived through the dot com bubble, I regret not soaking up more of the free and heavily subsidized stuff back then. Trying not to miss out this time. All this compute available for free or below cost won't last too much longer...

> which presumably hasn't done a fresh pre-training over the web

What makes you think that?

> Did they figure out how to do more incremental knowledge updates somehow?

It's simple. You take the existing model and continue pretraining with newly collected data.

Not an expert in LLM benchmarks, but I generally I think of benchmarks as being good particularly for measuring usefulness for certain usecases. Even if measuring LLMs is not as straightforward as, say, read/write speeds when comparing different SSDs, if a certain model's responses are consistently measured as being higher quality / more useful, surely that means something, right?

Do you have a better way to measure LLMs? Measurement implies quantitative evaluation... which is the same as benchmarks.

Garlic will be released 2026Q1.

Containment breach is going to occur from a disgruntled customer convincing the customer service bot it needs to get a hold of a supervisor

Yeah theres no wall on this. It will be able to mimic all of human behavior given proper data.

it was only about 2-3 weeks when several HNers told me "nah you better re-check your code", when I explained I have over 2 decades xp of coding, yet have not manually edited code (in memory) for the last 6 or so months, whilst performing daily 12 hour daily vibe code seshes

Ok so why isn’t there mass lay offs ensuing right now?

Also, did they mention these features? I was looking out for it but got to the end and missed it.

(No, I just looked again and the new features listed are around verbosity, thinking level and the tool stuff rather than memory or knowledge.)

thanks for letting us know.

I'm quite sad about the S-curve hitting us hard in the transformers. For a short period, we had the excitement of "ooh if GPT-3.5 is so good, GPT-4 is going to be amazing! ooh GPT-4 has sparks of AGI!" But now we're back to version inflation for inconsequential gains.

Are you using 5.1 Thinking? I tended to prefer Claude before this model.

I use models based on the task. They still seem specialized and better at specific tasks. If I have a question I tend to go to it. If I need code, I tend to go to Claude (Code).

I go to ChatGPT for questions I have because I value an accurate answer over a quick answer and, in my experience, it tends to give me more accurate answers because of its (over) willingness to go to the web for search results and question its instincts. Claude is much more likely to make an assumption and its search patterns aren't as thorough. The slow answers don't bother me because it's an expectation I have for how I use it and they've made that use case work really well with background processing and notifications.

This isn't the announcement, it's the developer docs intro page to the model - https://openai.com/index/introducing-gpt-5-2/. Still doesn't answer cross-comparison, but at least has benchmark metrics they want to show off.