...the evidence, as best I can tell from the tweet, is that they asked Claude what effort level it was set to. But how would the model even know that?

Not convinced here.

▲

ranie93 4 hours ago | parent | next [-]

I don’t know if this is still the case but while using Copilot if you looked through the chain-of-thought output you would see it reasoning about a “budget”. i.e. “since I’m close to the session budget I should…”. So it could be possible

▲

jerbear4328 4 hours ago | parent | prev | next [-]

Effort level is actually controlled entirely by system prompt (as I understand it, the model is trained on that format but still), so actually this is a valid way to check I think

▲

Wowfunhappy 4 hours ago | parent [-]

I know that's true for Qwen but I don't think most models work that way?

	▲	wren6991 4 hours ago \| parent [-]
		OpenAI models also work this way, as evidenced by full cache blowout when changing reasoning level. Every single open-weight model I've seen also works this way (your "reasoning_effort" argument just changes a small section of the system prompt in the chat template). I would have to see some evidence to believe Anthropic were doing anything different.

▲

varispeed 4 hours ago | parent | prev [-]

Then model can say it's Opus, but really it is some old Sonnet. This seems to be happening less often, but some weeks ago I had to give models some test problems to gauge whether I am getting Opus or something knee-capped.

The problem is that Anthropic seems to be getting away with selling one thing and delivering another. You pay for Opus, you get something else etc.

	▲	matltc 3 hours ago \| parent [-]
		Saw this in npx ccusage@latest claude output. Had only used opus but showed sonnet. Can't remember if the jsonl retains which model is doing what, but meh