| ▲ | Astra 6.2 and Sol 6.2 flop at Simple Bench(simple-bench.com) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 70 points by algoth1 3 hours ago | 54 comments | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | 0xbadc0de5 an hour ago | parent | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2. I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | nsingh2 13 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics. However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic". All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | ricardobeat an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models? Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | davidmurdoch an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | martinjc an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | algoth1 3 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | hoppp an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
All the new Gemini models perform better on this than Opus 5.5 Is that something that can be noticed during real life use? I really like opus 5.5, It's perfect and I haven't tried the new gemini models. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | goodcake3963 an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | TrackerFF an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Makes me wonder if people are getting different models? So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | Topfi 33 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Doesn't surprise me that other benchmarks are seeing regressions at this stage. Every OpenAI model starting with 5.5 has had noticeable issues in my evals that prior releases did not. First minor and specific to compaction, but with each release it got more severe. 6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait. There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case. To give some examples: The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back: > message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation. I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness and unsuitability. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me. Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty: > I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list. > Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one. I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great. Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains. I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet. And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between. As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguard public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now). Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | cheema33 3 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Simple Bench is mostly "trick questions". A model that is good at answering trick questions may not be great at your workflow. My girlfriend is good at answering trick questions. But isn't very good at software development. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | Culonavirus an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
6.2 is for sure the worst received update OpenAI has ever released. What happened lol. I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | loveparade 15 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The codex models are in the worst state they've ever been in. I used to be a huge codex fan, I wonder what happened. These models can't even complete simple tasks end-to-end anymore. Not only do they write super sloppy code compared to Claude, but now they just tend to stop mid-task like "the natural next step is XYZ" or "i found XYZ to be an issue but there is an easy fix" - yeah then why didn't you do it? Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | arthur-st an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | delichon 43 minutes ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). We could create a utopia if we could screen voters for linguistic adversarial robustness. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | bieganski an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
I tried taking the test, but ultimately stopped at Question 4: There are two sisters, Amy who always speaks mistruths and Sam who always lies. :) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | varispeed an hour ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | xqcgrek2 an hour ago | parent | prev [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
As the kids say, OpenAI is cooked. Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||