| ▲ | andai 3 hours ago | ||||||||||||||||
Would be interesting to see Sonnet (4.6*). It's fair bit smaller than Opus but scores pretty high on common sense, subjectively. I'm also curious about Haiku, though I don't expect it to do great. -- EDIT: Opus 4.6 Extended Reasoning > Walk it over. 50 meters is barely a minute on foot, and you'll need to be right there at the car anyway to guide it through or dry it off. Drive home after. Weird since the author says it succeeded for them on 10/10 runs. I'm using it in the app, with memory enabled. Maybe the hidden pre-prompts from the app are messing it up? I tested Sonnet 4.5 first, which answered incorrectly.. maybe the Claude app's memory system is auto-injecting it into the new context (that's how one of the memory systems works, injects relevant fragments of previous chats invisibly into the prompt). i.e. maybe Opus got the garbage response auto-injected from the memory feature, and it messed up its reasoning? That's the only thing I can think of... -- EDIT 2: Disabled memories. Didn't help. But disabling the biographical information too, gives: >Opus 4.6 Extended Reasoning >Drive it — the whole point is to get the car there! -- EDIT 3: Yeah, re-enabling the bio or memories, both make it stupid. Sad! Would be interesting to see if other pre-prompts (e.g. random Wikipedia articles) have an effect on performance. I suspect some types of pre-prompts may actually boost it. | |||||||||||||||||
| ▲ | stratos123 2 minutes ago | parent | next [-] | ||||||||||||||||
Interesting. I wonder if that's related to the phenomenon mentioned in the Opus 4.6 model card[1], where increased reasoning effort leads to 4.6 overthinking and convincing itself of the wrong answer on many questions. It seems to be unique to 4.6; I guess they fried it a bit too much during RL training. | |||||||||||||||||
| ▲ | Ethee 2 hours ago | parent | prev | next [-] | ||||||||||||||||
I tested this with Opus the day 4.6 came out and it failed then, still fails now. There were a lot of jokes I've seen related to some people getting a 'dumber' model, and while there's probably some grain of truth to that I pay for their highest subscription tier so at the very least I can tell you it's not a pay gate issue. | |||||||||||||||||
| ▲ | felix089 3 hours ago | parent | prev [-] | ||||||||||||||||
You mean Sonnet 4.6? I ran 9 claude models including Haiku, swipe through the gallery in the link to see their responses. | |||||||||||||||||
| |||||||||||||||||