Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat every clever adaptive scheme I measured.

▲

edg5000 2 days ago | parent | next [-]

Yes, but you could have an uncached dynamic tail, this would not kill the cache though. And you could still periodically kill the cache, still keeping the same prefix every N requests. But it's questionable what all this extra complexity delivers.

I found that now that the models are better, at least Codex compaction with GPT 5.5+ xhigh is almost unnoticible. So the answer is: just let the context fill up, don't prune much, the model is smart enough. When you're out of space, have the model summarize and continue from that point. A smart model will know the information pieces it needs to re-read after compaction; it won't make incorrect guessed when some details got lost.

▲

noisy_boy 2 days ago | parent [-]

That's what I have been doing with Cursor with auto mode. Let the model manage the cache through compaction but staying focused on the topic. It has been quite efficient cost wise and quality wise too. It found issuesbat cheaper cost that Sonnet 4.6 medium effort had missed while costing more. I would like to repeat this test with higher tier Anthropic models like Opus effort medium to see the results.

▲

edg5000 2 days ago | parent [-]

With a coding plan, there is no need to use lower quality models right (GPT 5.6, Fable 5). Only with some kind of batch job (processing PDFs) I could imagine not having enough usage with a 100 or 200 USD coding plan. It sounds like you're on API? I'm dying to know why, since it seems quite common, yet it's about ten times the cost. Sounds like a terrible deal. What am I missing?

	▲	noisy_boy 2 days ago \| parent [-]
		What I said was specific to work and it is not entirely clear to me why the cost goes up so much the moment I switch to medium with just Sonnet 4.6 (newer models would be even more I guess). There is a monetary budget allocated and I've blown it easily in the past with Claude. Cursor has been a life saver.

▲

stilesja a day ago | parent | prev | next [-]

I have a process generating trivia questions for my app and by passing in my existing dataset each time as a cached exclusion list, the output tokens it save more than pay for it.

▲

deemwar 18 hours ago | parent | prev | next [-]

[flagged]

▲

bkotrys 2 days ago | parent | prev [-]

[dead]