Remix.run Logo
zone411 2 days ago

I've benchmarked it on the Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/):

The high-reasoning version of GPT-5.2 improves on GPT-5.1: 69.9 → 77.9.

The medium-reasoning version also improves: 62.7 → 72.1.

The no-reasoning version also improves: 22.1 → 27.5.

Gemini 3 Pro and Grok 4.1 Fast Reasoning still score higher.

Donald 2 days ago | parent | next [-]

Gemini 3 Pro Preview gets 96.8% on the same benchmark? That's impressive

capitainenemo 2 days ago | parent | next [-]

And performs very well on the latest 100 puzzles too, so isn't just learning the data set (unless I guess they routinely index this repo).

I wonder how well AIs would do at bracket city. I tried gemini on it and was underwhelmed. It made a lot of terrible connections and often bled data from one level into the next.

wooger a day ago | parent [-]

> unless I guess they routinely index this repo

This sounds like exactly the kind of thing any tech company would do when confronted with a competitive benchmark.

rsanek a day ago | parent [-]

I mean, the repo has <200 stars, it's not like it's so mainstream that you'd expect LLM makers to be watching it actively. If they wanted to game it, they could more easily do that in RL with synthetic data anyway.

bigyabai 2 days ago | parent | prev [-]

GPT-5.2 might be Google's best Gemini advertisement yet.

outside1234 2 days ago | parent [-]

Especially when you see the price

tikotus 2 days ago | parent | prev | next [-]

Here's someone else testing models on a daily logic puzzle (Clues by Sam): https://www.nicksypteras.com/blog/cbs-benchmark.html GPT 5 Pro was the winner already before in that test.

thanhhaimai 2 days ago | parent | next [-]

This link doesn't have Gemini 3 performance on it. Do you have an updated link with the new models?

dezgeg a day ago | parent [-]

I've also tried Gemini 3 for Clues by Sam and it can do really well, have not seen it make a single mistake even for Hard and Tricky ones. Haven't run it on too many puzzles though.

crapple8430 2 days ago | parent | prev [-]

GPT 5 Pro is a good 10x more expensive so it's an apples to oranges comparison.

fellowniusmonk 7 hours ago | parent | prev | next [-]

I think they are overfitting more, I'm seeing it perform worse on esoteric logic puzzles.

Bombthecat a day ago | parent | prev | next [-]

I would like to see a cost per percent or so row. I feel like grok would beat them all

scrollop 2 days ago | parent | prev [-]

Why no grok 4.1 reasoning?

sanex 2 days ago | parent [-]

Do people other than Elon fans use grok? Honest question. I've never tried it.

buu700 a day ago | parent | next [-]

I use Grok pretty heavily, and Elon doesn't factor into it any more than Sam and Sundar do when I use GPT and Gemini. A few use cases where it really shines:

* Research and planning

* Writing complex isolated modules, particularly when the task depends on using a third-party API correctly (or even choosing an API/library at its own discretion)

* Reasoning through complicated logic, particularly in cases that benefit from its eagerness to throw a ton of inference at problems where other LLMs might give a shallower or less accurate answer without more prodding

I'll often fire off an off-the-cuff message from my phone to have Grok research some obscure topic that involves finding very specific data and crunching a bunch of numbers, or write a script for some random thing that I would previously never have bothered to spend time automating, and it'll churn for ~5 minutes on reasoning before giving me exactly what I wanted with few or no mistakes.

As far as development, I personally get a lot of mileage out of collaborating with Grok and Gemini on planning/architecture/specs and coding with GPT. (I've stopped using Claude since GPT seems interchangeable at lower cost.)

For reference, I'm only referring to the Grok chatbot right now. I've never actually tried Grok through agentic coding tooling.

wdroz a day ago | parent | prev | next [-]

Unlike openai, you can use the latest grok models without verifying your organization and giving your ID.

sz4kerto a day ago | parent | prev | next [-]

I'm using Gemini in general, but Grok too. That's because sometimes Gemini Thinking is too slow, but Fast can get confused a lot. Grok strikes a nice balance between being quite smart (not Gemini 3 Pro level, but close) and very fast.

jbm a day ago | parent | prev | next [-]

I use a few AIs together to examine the same code base. I find Grok better than some of the Chinese ones I've used, but it isn't in the same league as Claude or Codex.

mac-attack a day ago | parent | prev | next [-]

I can't understand why people would trust a CEO that regularly lies about product timelines, product features, his own personal life, etc. And that's before politicizing his entire kingdom by literally becoming a part of government and one of the larger donations of the current administration.

delaminator a day ago | parent | next [-]

You’re not narrowing it down.

lkjdsklf a day ago | parent | prev | next [-]

If we stopped using products of every company that had a CEO that lied about their products, we’d all be sitting in caves staring at the dirt

fatata123 a day ago | parent | prev [-]

Because not everyone makes their decisions through the prism of politics

rsanek a day ago | parent | prev | next [-]

it's the biggest model on OpenRouter, even if you exclude free tier usage https://openrouter.ai/state-of-ai

irthomasthomas a day ago | parent [-]

Roleplay is the largest use-case on openrouter.

19 hours ago | parent [-]
[deleted]
bumling a day ago | parent | prev | next [-]

I dislike Musk, and use Grok. I find it most useful for analyzing text to help check if there's anything I've missed in my own reading. Having it built in to Twitter is convenient and it has a generous free tier.

scrollop a day ago | parent | prev | next [-]

I hate the guy, however grok scores high on arc-2 so it would be silly to not at least rank it.

ralusek a day ago | parent | prev | next [-]

Only thing I use grok for is if there is a current event/meme that I keep seeing referenced and I don't understand, it's good at pulling from tweets

fatata123 a day ago | parent | prev [-]

[dead]