The benchmarks are impressive, but it's comparing to last generation models (Opus 4.5 and GPT-5.2). The competitor models are new, but they would have easily had enough time to re-run the benchmarks and update the press release by now.

Although it doesn't really matter much. All of the open weights models lately come with impressive benchmarks but then don't perform as well as expected in actual use. There's clearly some benchmaxxing going on.

▲

dongobread 5 hours ago | parent | next [-]

What a strangely hostile statement on an open weight model. Running like 20 benchmark evaluations isn't trivial by itself, and even updating visuals and press statements can take a few days at a tech company. It's literally been 5 days since this "new generation" of models released. GPT-5.3(-codex) can't even be called via API, so it's impossible to test for some benchmarks.

I notice the people who endlessly praise closed-source models never actually USE open weight models, or assume their drop-in prompting methods and workflow will just work for other model families. Especially true for SWEs who used Claude Code first and now think every other model is horrible because they're ONLY used to prompting Claude. It's quite scary to see how people develop this level of worship for a proprietary product that is openly distrusting of users. I am not saying this is true or not of the parent poster, but something I notice in general.

As someone who uses GLM-4.7 a good bit, it's easily at Sonnet 4.5 tier - have not tried GLM-5 but it would be surprising if it wasn't at Opus 4.5 level given the massive parameter increase.

	▲	apimade 3 hours ago \| parent \| next [-]
		Isn’t trivial? How is it not completely automated at this point?
	▲	maxdo 2 hours ago \| parent \| prev [-]
		but even opus 4.5 is history now, codex-5-3 and opus 4.6 are one more step forward. The opus itself caused paradigm shift, from writing code with AI, to ai is writing code with human. open weight models are not there at all yet.

▲

InsideOutSanta 7 hours ago | parent | prev | next [-]

> it's comparing to last generation models (Opus 4.5 and GPT-5.2).

If it's anywhere close to those models, I couldn't possibly be happier. Going from GLM-4.7 to something comparable to 4.5 or 5.2 would be an absolutely crazy improvement.

▲

Aurornis 6 hours ago | parent [-]

> Going from GLM-4.7 to something comparable to 4.5 or 5.2 would be an absolutely crazy improvement.

Before you get too excited, GLM-4.7 outperformed Opus 4.5 on some benchmarks too - https://www.cerebras.ai/blog/glm-4-7 See the LiveCodeBench comparison

The benchmarks of the open weights models are always more impressive than the performance. Everyone is competing for attention and market share so the incentives to benchmaxx are out of control.

▲

InsideOutSanta 6 hours ago | parent | next [-]

Sure. My sole point is that calling Opus 4.5 and GPT-5.2 "last generation models" is discounting how good they are. In fact, in my experience, Opus 4.6 isn't much of an improvement over 4.5 for agentic coding.

I'm not immediately discounting Z.ai's claims because they showed with GLM-4.7 that they can do quite a lot with very little. And Kimi K2.5 is genuinely a great model, so it's possible for Chinese open-weight models to compete with proprietary high-end American models.

▲

GorbachevyChase 5 hours ago | parent | next [-]

From a user perspective, I would consider Opus 4.6 somewhat of a regression. You can exhaust your the five hour limit in less than half an hour on, and I used up the weekly limit in just two days. The outputs did not feel significantly better than Opus 4.5 and that only feels smarter than Sonnet by degrees. This is running a single session on a pro plan. I don’t get paid to program, so API cost matter to me. The experience was irritating enough to make me start looking for an alternative, and maybe GLM is the way to go for hobby users.

▲

Aurornis 6 hours ago | parent | prev [-]

I think there are two types of people in these conversations:

Those of us who just want to get work done don't care about comparisons to old models, we just want to know what's good right now. Issuing a press release comparing to old models when they had enough time to re-run the benchmarks and update the imagery is a calculated move where they hope readers won't notice.

There's another type of discussion where some just want to talk about how impressive it is that a model came close to some other model. I think that's interesting, too, but less so when the models are so big that I can't run them locally anyway. It's useful for making purchasing decisions for someone trying to keep token costs as low as possible, but for actual coding work I've never found it useful to use anything other than the best available hosted models at the time.

	▲	buu700 4 hours ago \| parent \| next [-]
		It's high-interest to me because open models are the ultimate backstop. If the SOTA hosted models all suddenly blow up or ban me, open models mitigate the consequence from "catastrophe" to "no more than six to nine months of regression". The idea that I could run a ~GPT-5-class model on my own hardware (given sufficient capex) or cloud hardware under my control is awesome.
	▲	ffsm8 6 hours ago \| parent \| prev [-]
		For the record, opus 4.6 was released less then a week ago. That you think corporations are anything close to quick enough to update their communications on public releases like this only shows that you've never worked in corporate

▲

miroljub 5 hours ago | parent | prev | next [-]

Yeah, I'm sure closed source model vendors are doing everything within their power to dumb down benchmarks, so they can look like underdogs and play a pity game against open weight models.

Let's have a serious discussion. Just because Claude PR department coined the term benchmaxxing, we we should not be using it unless they shell out some serious monetes.

▲

KronisLV an hour ago | parent | prev [-]

I still enjoy using GLM 4.7 on Cerebras because of the speed you can get there and the frankly crazy amount of tokens they give you. Before that, 4.6 messed up file edits in OpenCode and VSC plugins more frequently, 4.7 is way more dependable but still has some issues with Python indentation and some partial edits sometimes (might also be tooling issue, e.g. using \ vs / as file separators in tool calls too) - but the quality of the output went up nicely!

I hope GLM 5 will also be available on Cerebras, since for the low-medium complexity work that's my go to, with Codex and Claude Code and Gemini CLI being nice for the more complex tasks.

▲

dust42 6 hours ago | parent | prev | next [-]

They are all just token generators without any intelligence. There is so little difference nowadays that I think in a blind test nobody will be able to differentiate the models - whether open source or closed source.

Today's meme was this question: "The car wash is only 50 meters from my house. I want to get my car washed, should I drive there or walk?"

Here is Claude's answer just right now:

"Walk! At only 50 meters (about 150 feet), it would take you less than a minute to walk there. Driving such a short distance would actually be less convenient - by the time you get in the car, start it, drive, and park, you could already be there on foot. Plus, you'd save fuel and your car would be right there waiting for you after the wash is done."

Here is ChatGPT, also right now:

"Since it’s only *50 meters* (about 160 feet), walking makes a lot more sense.

Here’s why:

* *It’s extremely close* — you’ll get there in under a minute. * *Driving would actually be less efficient* — by the time you start the car, move it, and park, you’re already there. * *Cold starts aren’t great for your engine* — very short drives don’t let the engine warm up properly. * *It saves fuel and emissions*, even if it’s a tiny amount. * You avoid the hassle of maneuvering such a short distance.

The only time driving might make sense is if:

* The weather is terrible (heavy rain, snow, extreme cold). * You have mobility issues. * The car wash requires you to stay in the vehicle and you don’t want to leave it there.

Otherwise, this is a perfect “walk it over” situation. "

Please save us all that talk about frontier and SOTA and that only the closedAI models are any good and the others are all so bad and benchmaxxed. For most purposes a Toyota is just as good as a BMW or a Merc or whatever luxury brand tickles your fancy. Even worse, the lastest 80B Qwen Next is not far from Opus 4.6 but runs on my laptop.

▲

cherryteastain 6 hours ago | parent | next [-]

Gemini 3 Flash ("Fast" in the web app) did not have trouble with this:

Since you presumably want the car to be clean, the answer depends entirely on how the car is getting washed. The Breakdown If it’s a Professional Car Wash: You’ll need to drive. Bringing the car to the wash is generally the prerequisite for them washing it. Plus, walking 50 meters just to ask them to come pick up a car that is visible from their front door might result in some very confused looks from the staff. If it’s a Self-Service Pressure Washer: Again, drive. Dragging a 1,500kg vehicle 50 meters by hand is an incredible workout, but probably not the "quick chore" you had in mind. If you are hiring a Mobile Detailer: Walk. Stroll over there, tell them where you're parked, and walk back home to watch TV while they do the work in your driveway.

▲

j-bos 6 hours ago | parent | prev | next [-]

GLM 4.7: "You should drive.

Since the goal is to get your car washed, the car needs to be at the car wash. If you walk, you will arrive at the car wash, but your car will still be sitting at home"

Are you sure that question is from this year?

▲

vardalab 3 hours ago | parent | prev | next [-]

This is really just a meme. People don't know how to use these tools.

Here is the response from Gpt-5.2 using my default custom instructions in the mac desktop app.

OBJECTIVE: Decide whether to drive or walk to a car wash ~50 meters from home, given typical constraints (car must be present for wash).

APPROACH: Use common car-wash workflows + short-distance driving considerations (warm engine, time, parking/queue). No web sources needed (stable, local decision).

OUTPUT: • Default choice: Drive. A car wash generally requires the car on-site, and 50 meters of driving is trivial. • When walking makes sense: • You’re going to drop the car off (hand-wash/detail) and pick it up later. • You want to avoid moving the car now (ice, tight parking, you’ve been drinking, etc.) and someone else will handle the car. • Micro-guidance: • If it’s an automatic wash you stay in, drive. • If it’s detail/hand wash with a wait, drive there, then walk home (or just wait).

CONFIDENCE: 92/100 Assumptions: It’s a normal car wash that requires your car to be there; conditions are safe to drive 50 meters.

▲

reliabilityguy 2 hours ago | parent | next [-]

Your objective has explicit instruction that car has to be present for a wash. Quite a difference from the original phrasing where the model has to figure it out.

▲

J_cst 2 hours ago | parent | next [-]

That's the answer of his LLM which has decomposed the question and built the answer following the op prompt obviously. I think you didn't get it.

▲

bwat49 2 hours ago | parent | prev [-]

> Your objective has explicit instruction that car has to be present for a wash.

Which is exactly how you're supposed to prompt an LLM, is the fact that giving a vague prompt gives poor results really suprising?

	▲	reliabilityguy 2 hours ago \| parent [-]
		In this case, with such a simple task, why even bother to prompt it? The whole idea of this question is to show that pretty often implicit assumptions are not discovered by the LLM.

▲

pegasus an hour ago | parent | prev | next [-]

Interesting, what were the instructions if you don't mind sharing?

▲

sph 2 hours ago | parent | prev [-]

"You're holding it wrong."

▲

haute_cuisine 6 hours ago | parent | prev | next [-]

Doesn't seem to be the case, gpt 5.2 thinking replies: To get the car washed, the car has to be at the car wash — so unless you’re planning to push it like a shopping cart, you’ll need to drive it those 50 meters.

▲

Aurornis 6 hours ago | parent | prev | next [-]

If you're asking simple riddles, you shouldn't be paying for SOTA frontier models with long context.

This is a silly test for the big coding models.

This is like saying "all calculators are the same, nobody needs a TI-89!" and then adding 1+2 on a pocket calculator to prove your point.

▲

Balinares 4 hours ago | parent | next [-]

I find it's a great test, actually. There are lots of "should I take the car" decisions in putting together software that's supposed to do things, and with poor judgement in how the things should be done, you typically end up with the software equivalent of a Rube-Goldberg machine that harnesses elephants to your car and uses mice to scare the elephants toward the car wash while you walk. After all, it's a short distance, isn't it?

▲

grey-area 6 hours ago | parent | prev [-]

No it’s like having a calculator which is unable to perform simple arithmetic, but lots of people think it is amazing and sentient and want to talk about that instead of why it can’t add 2 + 2.

	▲	viraptor 2 hours ago \| parent [-]
		We know why it's not going to do precise math and why you can have better experience asking for an app solving the math problem you want. There's no point talking about it - it's documented in many places for people who are actually interested.

▲

Scene_Cast2 6 hours ago | parent | prev | next [-]

I just ran this with Gemini 3 Pro, Opus 4.6, and Grok 4 (the models I personally find the smartest for my work). All three answered correctly.

▲

miroljub 5 hours ago | parent [-]

They had plenty of time to update their system prompts so they don't be embarrassed.

I noticed whenever such meme comes out, if you check immediately you can reproduce it yourself, but after a free hours it's already updated.

▲

viraptor 3 hours ago | parent | next [-]

I think you're seriously underestimating how much effort the fine tuning at their scale takes and what impact it has. They don't pack every edge case into the system prompt either. It's not like they update the model every few hours or even care about memes. If they seriously did, they'd force-delegate spelling questions to tool calls.

▲

suzzer99 3 hours ago | parent [-]

Could it be the model is constantly searching its own name for memes, or checking common places like HN and updating accordingly? I have no idea how real-time these things are, just asking.

	▲	viraptor 2 hours ago \| parent [-]
		The model doesn't do anything on its own. And it's usually months in between new model snapshots.

▲

selcuka an hour ago | parent | prev | next [-]

I tested it on Claude and only Opus 4.6 answers it correctly. Haiku and Sonnet can't and Opus 4.5's reply is unintelligible. The would've updated the system prompts for all models.

▲

qiine 4 hours ago | parent | prev | next [-]

The road to AGI is weirder than anticipated

▲

UrineSqueegee 3 hours ago | parent | prev [-]

thats not how it works

▲

qwytw 3 hours ago | parent | prev | next [-]

Some snippets from Kimi's 2.5 answer:

"" [...] Since you need to get your car washed, you have to bring the car to the car wash—walking there without the vehicle won't accomplish your goal [...] If it's a self-service wash, you could theoretically push the car 50 meters if it's safe and flat (unusual, but possible) [..] Consider whether you really need that specific car wash, or if a mobile detailing service might come to you [...] """

Which seems slightly (unintentionally) funny.

But to be fair all the Gemini (including flash) and GPT models I tried did understand the quesiton.

▲

uncharted9 4 hours ago | parent | prev | next [-]

I tried this prompt with all the major AI models that I know and have installed, and only GLM from Z.ai and Gemini 3 Flash could answer it. Even in that case, Gemini 3 Flash gave a bit more nuanced answer, but every other model like Claude Opus 4.5, Chat GPT, Grok - everything failed.

▲

bonoboTP 6 hours ago | parent | prev | next [-]

It's unclear where the car is currently from your phrasing. If you add that the car is in your garage, it says you'll need to drive to get the car into the wash.

▲

RohMin 4 hours ago | parent [-]

Do you think the average person would need this sort of clarification? How many of us would have recommended to walk?

	▲	bonoboTP 3 hours ago \| parent [-]
		Do you think this is a fundamentally unbridge-able limitation of LLMs? Do you know where we were just a year ago? Can you imagine that this will get better with upcoming releases? It's like when Gary Marcus was confidently stating that AI (at least current paradigm) will never be able to generate an image of a horse riding an astronaut. (Or full wineglasses or arbitrary clocks).

▲

king_phil 6 hours ago | parent | prev | next [-]

Gemini 3 Pro:

This is a classic logistical puzzle!

Unless you have a very unique way of carrying your vehicle, you should definitely drive.

If you walk there, you'll arrive at the car wash, but your car will still be dirty back at your house. You need to take the car with you to get it washed.

Would you like me to check the weather forecast for $mytown to see if it's a good day for a car wash?

	▲	interloxia 5 hours ago \| parent [-]
		For me, various forms of Gemini respond with "Unless you are planning on carrying the car there" which I find to be just sassy enough to be amusing.

▲

smokel 6 hours ago | parent | prev | next [-]

> They are all just token generators without any intelligence.

Maybe, but "intelligence" doesn't have a clear, agreed definition. And calling them "just token generators" skips over how complex that generation actually is.

▲

reliabilityguy 2 hours ago | parent | prev | next [-]

Gemini Pro:

You should definitely drive.

If you walk there, your car will still be dirty back at your house! Since the goal is to get the car washed, you have to take it with you.

PS fantastic question!

▲

esafak 6 hours ago | parent | prev | next [-]

This is a great comment. The frontier labs ought to be embarrassed they can't get this right. A human would not make the same mistake, after thinking for a second.

▲

Hammershaft 5 hours ago | parent | prev | next [-]

Claude 4.6 got it first try:

"You’ll want to drive — you need the car at the car wash for them to wash it!

Even though it’s just 50 meters, the car has to be there. Enjoy the fresh clean car on the short drive back! "

▲

izucken 3 hours ago | parent | prev | next [-]

Tsk tsk thats adversarial prompting. Hostile even. And thats goalposting which is totally not cool, man.

▲

rockinghigh 6 hours ago | parent | prev | next [-]

How is this riddle relevant to a coding model?

	▲	esafak 6 hours ago \| parent [-]
		It's not a coding model. Go to https://chat.z.ai/ and you'll see it is presented as a generalist.

▲

UrineSqueegee 3 hours ago | parent | prev | next [-]

this is a bot comment or just ragebait

▲

zozbot234 6 hours ago | parent | prev [-]

1. Analyze the User's Request:

Distance: 50 meters (very short).

Goal: Get the car washed.

Decision: Drive or walk?

2. Evaluate the Options:

Option A: Walk

Pros: Healthy, environmentally friendly, no need to find parking (though it's close), easy to chat with neighbors.

Cons: The car is left at the car wash. The user can't "walk" the car to the car wash. The car must be moved to the car wash to be washed.

Option B: Drive

Pros: Moves the car to the location where the service is performed. It's the only logical way to get the car washed.

Cons: Driving 50 meters is technically bad for a cold engine (short trips), uses a negligible amount of fuel, might seem lazy.

3. Identify the Logical Flaw/Humor:

The user is asking if they should walk to get their car washed.

Unless the car wash involves someone coming to their house (mobile wash) or carrying the car, the car must be at the car wash.

Walking to the car wash leaves the car at home. Walking back from the car wash leaves the car at the car wash.

Therefore, the user must drive the car to the car wash.

4. Formulate the Answer:

Direct Answer: You should drive.

Reasoning: The car needs to be at the car wash to be washed. You cannot walk the car there.

Nuance/Context: 50 meters is a very short distance. Driving a car such a short distance is generally not great for the engine (doesn't allow oil to circulate/warm up properly), but it is necessary in this specific case.

Alternative (if applicable): Is there a mobile wash option? If so, walking isn't even needed. But assuming a standard car wash bay: Drive.

5. Drafting the response (Internal Monologue/Trial Phrases):

Draft 1: You have to drive. You can't wash the car if it's at your house and you are at the car wash.

Draft 2: You should drive. Since the car needs to be washed, it needs to be at the car wash. Walking there doesn't help get the car clean.

Refining for clarity and wit: Point out the obvious - the car is the object being washed. It needs to move.

Final Polish (similar to the provided good output):

State clearly: You should drive.

Explain why: The car needs to be there.

Add a practical note: Driving 50 meters is a bit silly for engine health, but unavoidable here.

Consider the scenario of "walking back": Maybe the user meant "walk back"? No, the prompt says "drive there or walk there". The intent is transportation to the location.

6. Constructing the final output:

"You should definitely drive."

"Why? Because the car wash cleans the car, so the car needs to be at the car wash. If you walk, the car stays at home."

"Additional thought: Driving 50 meters is technically a 'cold start' short trip which isn't ideal for the engine, but it's the only way to get the service done!"

Final Output Generation (matching the logic above).

You should drive.

Here is why:

1. The car needs to be there: You are getting the car washed, so the car must be at the car wash. You cannot walk the car there by yourself.

2. Efficiency: Driving 50 meters takes about 10 seconds. Walking 50 meters takes about a minute, but you would be leaving the car behind.

The only exception is if you are planning to get a mobile car wash (where they come to you) or if you are dropping the car off and walking back home while they clean it. But to simply "get there," you must drive.

(The above output is from GLM-5)

▲

throwup238 7 hours ago | parent | prev | next [-]

> Although it doesn't really matter much. All of the open weights models lately come with impressive benchmarks but then don't perform as well as expected in actual use. There's clearly some benchmaxxing going on.

Agreed. I think the problem is that while they can innovate at algorithms and training efficiency, the human part of RLHF just doesn't scale and they can't afford the massive amount of custom data created and purchased by the frontier labs.

IIRC it was the application of RLHF which solved a lot of the broken syntax generated by LLMs like unbalanced braces and I still see lots of these little problems in every open source model I try. I don't think I've seen broken syntax from the frontier models in over a year from Codex or Claude.

▲

algorithm314 7 hours ago | parent | next [-]

Can't they just run the output through a compiler to get feedback? Syntax errors seem easier to get right.

	▲	NitpickLawyer 6 hours ago \| parent \| next [-]
		The difference is in scaling. The top US labs have oom more compute available than chinese labs. The difference in general tasks is obvious once you use them. It used to be said that open models are ~6mo behind SotA a year go, but with the new RL paradigm, I'd say the gap is growing. With less compute they have to focus on narrow tasks, resort to poor man's distillation and that leads to models that show benchmaxxing behavior. That being said, this model is MIT licensed, so it's a net benefit regardless of being benchmaxxed or not.
	▲	rockinghigh 6 hours ago \| parent \| prev [-]
		They do. Pretty much all agentic models call linting, compiling and testing tools as part of their flow.

▲

ej88 7 hours ago | parent | prev [-]

the new meta is purchasing rl environments where models can be self-corrected (e.g. a compiler will error) after sft + rlhf ran into diminishing returns. although theres still lots of demand for "real world" data for actually economically valuable tasks

▲

miki123211 4 hours ago | parent | prev | next [-]

Anthropic, OpenAI and Google have real user data that they can use to influence their models. Chinese labs have benchmarks. Once you realize this, it's obvious why this is the case.

You can have self-hosted models. You can have models that improve based on your needs. You can't have both.

▲

viraptor 3 hours ago | parent [-]

zAI, minimax and Kimi have plenty of subscriber usage on their own platforms. They get real data just as well. Less or it maybe but it's there.

▲

Art9681 2 hours ago | parent [-]

I'm going to claim that the majority of those users are optimizing for cost and not correctness and therefore the quality of data collected from those sessions is questionable. If you're working on something of consequence, you're not using those platforms. If you're a tinkerer pinching pennies, sure.

	▲	cherryteastain 15 minutes ago \| parent \| next [-]
		ChatGPT, Gemini and Claude are banned in China. Chinese model providers are getting absolutely massive amounts of very valuable user feedback from users in China.
	▲	viraptor an hour ago \| parent \| prev [-]
		This is a weird dichotomy and I don't agree with it. You don't need to have bags of money to burn to work on serious things. You also can value correctness if you're poor.

▲

ionelaipatioaei 6 hours ago | parent | prev | next [-]

I think the only advantage that closed models have are the tools around them (claude code and codex). At this point if forced I could totally live with open models only if needed.

▲

evv 6 hours ago | parent | next [-]

The tooling is totally replicated in open source. OpenCode and Letta are two notable examples, but there are surely more. I'm hacking on one in the evenings.

OpenCode in particular has huge community support around it- possibly more than Claude Code.

	▲	ionelaipatioaei 6 hours ago \| parent \| next [-]
		I know, I use OpenCode daily but it still feels like it's missing something - codex in my opinion is way better at coding but I honestly feel like that's because OpenAI controls both the model and the harness so they're able to fine tune everything to work together much better.
	▲	Daviey 6 hours ago \| parent \| prev [-]
		It's there now, `opencode models --refresh`

▲

hamdingers 3 hours ago | parent | prev | next [-]

GLM works wonderfully with Claude, just have to set some environment variables and you're off to the races.

▲

quikoa 6 hours ago | parent | prev [-]

If tooling really is an advantage why isn't it possible to use the API with a subscription and save money?

	▲	ionelaipatioaei 6 hours ago \| parent [-]
		In my opinion it is because if you control both the model and the harness then you're able to tune everything to work together much better.

▲

cmrdporcupine 7 hours ago | parent | prev | next [-]

I tried GLM 5 by API earlier this morning and was impressed.

Particularly for tool use.

▲

yieldcrv 7 hours ago | parent | prev [-]

come on guys, you were using Opus 4.5 literally a week ago and don't even like 4.6

something that is at parity with Opus 4.5 can ship everything you did in the last 8 weeks, ya know... when 4.5 came out

just remember to put all of this in perspective, most of the engineers and people here haven't even noticed any of this stuff and if they have are too stubborn or policy constrained to use it - and the open source nature of the GLM series helps the policy constrained organizations since they can theoretically run it internally or on prem.

	▲	Aurornis 6 hours ago \| parent [-]
		> something that is at parity with Opus 4.5 You're assuming the conclusion The previous GLM-4.7 was also supposed to be better than Sonnet and even match or beat Opus 4.5 in some benchmarks ( https://www.cerebras.ai/blog/glm-4-7 ) but in real world use it didn't perform at that level. You can't read the benchmarks alone any more.