| ▲ | kelnos 6 hours ago |
| This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the context window that far anyway when working on a single task, or a series of tasks that are related enough to warrant the same context; more typical is anywhere between 200k and 600k or so.) I'm not saying that no one ever has this experience, but it's odd to me that some people see it so often that it warrants giving it a name. |
|
| ▲ | Bolwin 6 hours ago | parent | next [-] |
| I see this said often and find it insane given how many times I find opus models making basic recall mistakes at <100k tokens. Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer |
| |
| ▲ | eterm 6 hours ago | parent | next [-] | | 60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt. | | |
| ▲ | danielbln 6 hours ago | parent | next [-] | | Yeah 60k is ludicrous, I've barely seeded the context at that point and I don't see context related degradation until well into the 600-700k. | | |
| ▲ | qsera 5 hours ago | parent | next [-] | | In this thread: People tossing coins independently and fighting over the result they got. | | |
| ▲ | kuboble 3 hours ago | parent [-] | | No it's not. It seems that people have different workflows or repos, or memories or prompts or expectations. | | |
| ▲ | diab0lic 2 hours ago | parent [-] | | For what it’s worth, as a third party I read your and qsera’s comments as saying the same thing. |
|
| |
| ▲ | embedding-shape 5 hours ago | parent | prev | next [-] | | > I've barely seeded the context at that point I think that's issue, rather than 60K being small. Most of the actual edits/changes I request to codex are solved within 100-150K tokens, beyond 200K I'd definitively try to restart the session as soon as I could as all models are horrible once you get across ~20% of the total context size. And this is while working on +million LOC codebases. Problem I guess is that there is no solid and concrete evidence of this (to me [and others seemingly] obvious) degradation, but should be easy to prove, yet no one has time to sit down and show it :) But the likelihood of a model getting minor details wrong once you're above some magical threshold between 15-20%, seems to skyrocket, and I hit that issue sufficient amount of times that now my workflow is trying to prevent that. | |
| ▲ | rtpg 2 hours ago | parent | prev [-] | | what are y'all doing to hit that? Do you just not give it any pointers and let it churn away? What kind of context are you handing off? I routinely get claude to do things pretty decently and finish up easily in the 4-5 digit range of tokens. It seems to be doing the right kind of thing to not waste its time looking at 1000 files. |
| |
| ▲ | da_grift_shift 6 hours ago | parent | prev [-] | | >you might have some false memories or incorrect instructions in your CLAUDE.md "YOU'RE HOLDING IT WRONG!"
| | |
| |
| ▲ | nijave 2 hours ago | parent | prev | next [-] | | >making basic recall mistakes at <100k tokens. I usually see this when the context gets "tainted" as I call it. The model gets stuck on a bad path and there's no way to bring it back without clearing the context and starting again. Frequently it'll be something as small as 1 sentence of a prompt many messages ago. When cases like that happen, I reset the context and try to be explicit about assumptions and requirements to keep it off the "tainted" path. Other times it's actually useful and agents will do things they normally wouldn't do once the state is tainted. For instance, if you're testing a chat bot's ability to stay on topic, you can seed the context early with what you want it to do. It generally will refuse initially but later on in the conversation it will still silently take that seeded context into account almost "subconsciously" and become more likely to do the thing it originally refused. | |
| ▲ | CjHuber 5 hours ago | parent | prev | next [-] | | I'm always a bit confused when people say things like this. 60k token is often more than the initial context I feed the model with. And I don't think I ever had a productive session that began under 150k tokens. | | |
| ▲ | embedding-shape 5 hours ago | parent [-] | | Bit of what makes it so fun, our experiences seem to wildly differ! On one hand, you have experiences like yours, but then my own experience is that I never had a productive session when the scope grows beyond 150K tokens! If I needed 60K just as a starting context, I'd take that to mean the suggested change is way to large, and if the model cannot solve the entire thing within maybe 15-20% of the total context size, divide and conquer is needed otherwise there will be a lot of time wasted to patch things up when things are "completed". | | |
| ▲ | CjHuber 3 hours ago | parent [-] | | Yeah indeed it's very interesting. And the 60k initial context don't even contain the suggested change yet. For me if I don't do this the current models tend to fixate and local patches instead of tracing symbols and making a holistic model of what a change interacts with in the codebase |
|
| |
| ▲ | wg0 6 hours ago | parent | prev | next [-] | | Not specific to Opus but yes it would make mistakes. I usually try to keep context window under 10% | |
| ▲ | properbrew 6 hours ago | parent | prev [-] | | I hate to do the "you're holding it wrong" trope, but I think you might have something misconfigured somewhere unless you missed a 0, because just past 60k tokens is such a small context window to be seeing issue in. Do you have any old documentation that it's picking up and referencing? If you set all claude settings back to default do you see the same issue? |
|
|
| ▲ | arcanemachiner 6 hours ago | parent | prev | next [-] |
| Opus 4.6 was on drugs past 200k, I skipped 4.7, 4.8 did good up to ~350k, and Fable did great beyond 400k, in my limited testing. The quality does appear to be trending upwards. |
| |
|
| ▲ | tyleo 3 hours ago | parent | prev | next [-] |
| I often push past 300k or so and I’ve absolutely worked at 800k but it’s an observable problem. Large context windows can work depending on the problem but I do feel more effective biasing towards small ones <300k. |
|
| ▲ | fullstackchris 6 hours ago | parent | prev | next [-] |
| Thats another problem of this post, the author mentions Claude but not explicitely what models... 100k tokens "by lunch" is also not my finding, the newer models will hit that already right in the initial exploratory phase |
| |
| ▲ | arcanemachiner 6 hours ago | parent | next [-] | | Really depends on the project. | |
| ▲ | stavros 6 hours ago | parent | prev [-] | | I found "by lunch" odd too, but considering that Claude wrote the article, it's not going to know specifics. |
|
|
| ▲ | asd88 6 hours ago | parent | prev | next [-] |
| I’ve had similar experiences with Fable. 70%+ context used out of 1M, still sharp and no memory issues. |
|
| ▲ | csomar 6 hours ago | parent | prev | next [-] |
| I have a custom build command for a rust project (yarn build:lib) and my experience is 120k for GLM and roughly 200-300k for Opus. After that, they default to cargo build. |
| |
| ▲ | trapexit 6 hours ago | parent [-] | | My projects have specific build/verify steps as well, and after a certain point Claude forgets to run them. I’m going to try a “No brown M&Ms” hook to halt Claude if it tries to run the default command instead of the instructed commands from CLAUDE.md. Perhaps this will be a good signal that a compacted or fresh session is needed at that point to avoid mistakes. | | |
| ▲ | csomar an hour ago | parent [-] | | I mean, that’s basically the magic of the harness. The whole thing that skyrocketted the intelligence is that the harness (cli tool) prevent the LLM from editing the file before reading it. Can you imagine even a junior making such a mistake? |
|
|
|
| ▲ | cyanydeez 6 hours ago | parent | prev [-] |
| As the gamblers say at the poker table: If you can't figure out who the mark is when you site down... |
