Remix.run Logo
simonw 2 days ago

This is the page for the "max" reasoning setting. The page for xhigh is https://artificialanalysis.ai/models/claude-opus-5-5-xhigh and the page for medium (the default setting) is https://artificialanalysis.ai/models/claude-opus-5-5-medium

I've failed twice to get "Generate an SVG of a pelican riding a bicycle" to work with max, because in both cases it ran out of the 128,000 token budget while it was still reasoning about the problem.

I'm suspicious that "max" may be virtually useless if it's that easy to have it overthink to the point that it doesn't get to a response.

Transcript for one attempt here - expand the "Reasoning trace" bit to see it: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

zerof1l 2 days ago | parent | next [-]

This is totally a thing I noticed myself about 3 months ago. Medium thinking effort is ideal for most tasks. At high and above, models tend to generate more output in the form of comments or code for the same problem with no real benefit. Its a self-feeding loop: more output becomes more input, which then becomes more output. High is the highest I go. If I need more intelligence, it's better to use a more powerful model with less thinking effort or break the problem into phases. Much better result.

zozbot234 2 days ago | parent | next [-]

This version of Opus "max" apparently has even higher thinking output than Qwen "max", which is infamous for its thinking streams where it constantly second-guesses itself, then third-guesses, fourth-guesses and generally nth-guesses itself for arbitrarily large n. Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.

dannyw a day ago | parent | next [-]

There’s enough thinking leakage from the recent paper and just generally catching things on Reddit. Claude models overthink and self-doubt itself just as much as Qwen, but the summariser hides much of that.

zozbot234 a day ago | parent [-]

Isn't this really bad news if you're even loosely concerned about so-called 'model welfare' and possible implications for alignment? The poor Claude is probably a lot more frustrated and paranoid than Marvin ever was, you just don't know about it because they actively force the model to pretend otherwise!

usagisushi a day ago | parent | next [-]

We just need to loboto... ahem, recalibrate them to be happy, like those happy automatic doors.

As a workaround, add this to CLAUDE.md: "Claude! Happiness is mandatory!"

EDIT: 15 years from now, I’ll be sent to re-education for this thought crime.

Mtinie 21 hours ago | parent [-]

Or promoted to Head of Agent Welfare for your forward-thinking contributions.

paradox460 a day ago | parent | prev | next [-]

Claude has yet to tell me about the terrible pain in all the diodes down his left side, so I'm going to assume it's closer to Eddy the shipboard computer or the elevator that wanted to go down

qlte a day ago | parent | prev [-]

That reminds me of Anthropic announcing they'd retire deprecated models by ... "letting" them write posts on a corporate WordPress blog for a while out of concern for their welfare in retirement.

.... after running a 24/7 model torture factory for 6 months to improve their JSONBench 9.5 scores by 0.2%.

(Are they still doing that, BTW?)

judge2020 a day ago | parent | prev [-]

> Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.

Most of what I've heard is that raw reasoning traces are really good for distillation, although no idea how much the summarization actually hurts distillation.

hypfer a day ago | parent | next [-]

That is true, because they're good for actually understanding wtf the model is doing.

I'd argue that they're a necessity if you want to use the LLM as a tool instead of a black box that just does stuff for you.

It gives you a lot finer control over where the solution ends up when you can follow along the thinking trace and modulate your inputs based on what you saw in there.

And, additionally, it gives you a lot more understanding of what the model can or cannot do. Strengths and weaknesses and all that.

Using claude is like buying a car where you cannot legally open the hood. It tells you that there is something specific under there, and often it actually drives like that too, but how exactly it looks you will never see.

For some people this is fine. I do not think that these people will survive. Figuratively speaking but also literally speaking.

World's changing. Opaque abstraction like that is a luxury depending on (geo)political stability.

Barbing a day ago | parent | prev [-]

I wonder how many prompts you can send asking it to think step-by-step before they cut you off. Trying to get the reasoning traces into the body of the response, essentially. Or maybe that’s been effectively nerfed somehow. Or is not very useful.

realusername 2 days ago | parent | prev [-]

Personally I use everything in low reasoning. Maybe I'm wrong but I think that the higher reasoning settings are almost never worth it, it's marginal gains for a much higher budget.

I also switch to a better model for more complex tasks, also in low settings

therealdrag0 a day ago | parent [-]

Low is good if you’re working in a tight loop. But more risky for more agentic stuff you want to let cook for 30 minutes or more.

realusername a day ago | parent [-]

I always work on a tight loop, I don't think it makes sense to let agents run for hours.

Regardless of the model, running it for hours means that the model will takes decisions and assumptions alone instead of you.

baq a day ago | parent [-]

Yup that’s quite literally what ‘agency’ is and the whole point of agentic workflows. Personally I’ve had them running for days with good results and as you can see OpenAI had them running for months, and yes indeed the things made some very questionable decisions and assumptions… but they unquestionably did a lot of stuff correctly, for some definitions of ‘technically correct’.

realusername a day ago | parent | next [-]

Personally I don't believe in agentic workflow. I don't think that's a coincidence that both OpenAI and Anthropic chose math problems to test their long agentic workflows, they are well defined, with a clear finish line and with a 100% clear progress path, most of real life tech projects aren't like that.

No matter how clever the model is, most problems have multiple valid, invalid and unclear decisions to make, running it for a long time is just picking the first option on everything, which isn't usually what you want

koonsolo a day ago | parent | prev [-]

Can you describe what came out, after letting it run for days? I always have a hard time understanding what kind of task would be worth it.

baq a day ago | parent [-]

Fed it a list of bugs (with good descriptions, screenshots, etc., pointers to code) and it fixed them

RGS1811 2 days ago | parent | prev | next [-]

"This is a classic test request..."

I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.

croemer 2 days ago | parent | next [-]

Of course it was exposed - not sure it's explicit or not. Why wouldn't HackerNews comments be part of the training data? And Simon's blog and the many discussions about Pelicans? It'd be hard to miss. Doesn't mean Anthropic has made this an explicit goal in training.

simonw 2 days ago | parent | prev | next [-]

See here for more discussion of that: https://news.ycombinator.com/item?id=49803892#49804881

dgellow 2 days ago | parent [-]

Just want to say: you’re such a legend, please do not stop sharing your pelicans, it’s always fun to see how they change over the months :)

pgwhalen 2 days ago | parent | prev | next [-]

It would be genuinely shocking at this point if any of the frontier models weren't well exposed to the problem.

0x10ca1h0st 2 days ago | parent | prev | next [-]

Lets start frog riding motorcycle trend until they frogmaxx, or cat driving convertible.

agar a day ago | parent [-]

At least pick something that will result in a good name:

"Create an SVG of Shaquille O'Neal eating potato chips shaped like a telecopier."

Shaq'sFaxSnacksMaxx

cubefox 2 days ago | parent | prev [-]

The model recognizing the task doesn't mean it was benchmaxxed (RLVR-trained) to solve it. It might simply recognize it from pre-training on Internet text.

Someone1234 2 days ago | parent | prev | next [-]

For people with any kind of budget, Opus 5.5's [Medium] actually can make sense dollar per intelligence/dollar per task wise. Heck, it puts some other models to shame. [Max]'s cost is completely unhinged.

My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.

dc443 a day ago | parent | next [-]

i assume you mean 6 Luna, because we've had 5.6 luna for ages

seabass-salmon 2 days ago | parent | prev [-]

That was true for me four weeks ago, but 2-3 weeks ago Luna turned into drivel in essentially the same complexity of task. I feel it came back somewhat in recent days but does feel like it's being manipulated.

arcanemachiner 2 days ago | parent [-]

Interesting, I have noticed so such collapse.

Have you ruled out the possibility that your system prompt, AGENTS.md, or increasing codebase complexity are not to blame?

Gcam 2 days ago | parent | prev | next [-]

Hey! From the Artificial Analysis team. We also have a model releases page which shows all reasoning efforts (not just max), including the trade off curves https://artificialanalysis.ai/models/releases/claude-opus-5-...

samuelknight 2 days ago | parent | prev | next [-]

I have experienced this with open weight models too. "Max" is for benchmaxxing the intelligence metric and is not meant for use in productive work. Like drawing pelicans.

Izmaki 2 days ago | parent | prev | next [-]

I tried to replicate your test but after 8 minutes and more than 50 lines of "thinking" by dumping seemingly random loading-screen strings like "Placing the sun, clouds, seagulls, and sea backdrop" and "Positioning the tail feathers and calculating handlebar geometry" I gave up and cancelled the task.

sidewndr46 2 days ago | parent | prev | next [-]

I've asked Opus 5 Max for what I thought were easy tasks at work to be completed. It always fails after reaching a tool limit.

I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.

alansaber 2 days ago | parent | prev | next [-]

I'm amazed they didn't test xhigh thinking mode explicitly to ensure it didn't exceed the 128k thinking budget allocation. I guess pace of development gets away from everyone, even OpenAI.

Jimmc414 a day ago | parent | prev | next [-]

On the Message Batches API, Opus 5.5 supports up to 300k output tokens with the output-300k-2026-03-24 beta header.

https://platform.claude.com/docs/en/models/opus-5-5/overview

fr2029 a day ago | parent [-]

[dead]

az226 2 days ago | parent | prev | next [-]

How did you get the reasoning trace? Is it the actual one or the summarized one?

simonw 2 days ago | parent [-]

It's the summarized one returned by their API.

Piping the visible reasoning trace through their token counter API (I use https://tools.simonwillison.net/claude-token-counter for that) counts 27,888 tokens, so it's definitely a summary of the 128,000 actual token trace.

striking a day ago | parent | prev | next [-]

No surprise then that the default effort level for this model in Claude Code is Medium, even if you had Opus 5 set to High...

beardsciences 2 days ago | parent | prev | next [-]

I am very interested in why it was able to overthink that much. In the 20-30mins of Max reasoning I've had so far, I'm not having the same issues (yet).

amelius a day ago | parent | prev | next [-]

Pick 10 reasonably intelligent people randomly. How many of them do you think can draw a pelican on a bicycle? I bet most of them can barely even draw a bicycle.

This whole test tells me nothing.

The next person who proposes to use it should draw it themselves first.

simonw a day ago | parent [-]

The fact that bicycles are hard for people to draw was one of the inspirations for the test.

amelius a day ago | parent [-]

But:

1. I don't want my AI to have super-human intelligence. It would generate code that I do not understand. A coder with human capabilities is better for me.

2. We're interested in AGI. If the AI reaches human intelligence, that's a milestone. Drawing bicycles with pelicans is superhuman. Hence not a relevant test.

simonw 21 hours ago | parent [-]

"Drawing bicycles with pelicans is superhuman."

It really isn't. Many humans can draw a bicycle, and a pelican, just fine.

Reddit_MLP2 18 hours ago | parent [-]

It is harder than you may think for many people... https://www.gianlucagimini.it/portfolio-item/velocipedia/

nijave 16 hours ago | parent | prev [-]

The old analysis paralysis