Author here. Context for why I did this:

There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions idle and LRU can't distinguish a paused session from a dead one. I found the argument convincing and built a simulator to exploit it. Three separate mechanisms, all lost to plain radix-leaf LRU.

The reason turned out to be more useful than the policy. When I measured — policy-independently — where recompute actually comes from on 393 real Claude Code sessions, requests arriving after a gap longer than the 5-minute provider TTL account for 17.5% of it. Requests arriving within 10 seconds account for 33.1%. The dominant waste is tight tool loops whose 88k-token working sets exceed cache capacity, not sessions idling past a TTL. That's a capacity problem, and liveness prediction can't touch it.

▲

r_lee 10 hours ago | parent | next [-]

just curious, why the LLM writing even here?

it's just a bit disheartening to read Claude output for such a small comment like this.. it'd be great to read your own writings even if it's not as "perfect"

▲

pmarreck 9 hours ago | parent [-]

One side effect of extremely-accessible, high-quality English translation and, more importantly, English-grammar-and-idiosyncrasy-obeying AI, is that there will be more and more text that looks like this which comes from foreign, non-native-English countries.

Just FYI.

This will be especially true from non-English cultures where "avoiding shame" is high on the list of motivations.

Consider the upside, though: A much broader range of written perspectives written in high-quality, if slightly annoying, English.

I'm already seeing the benefit of this on X thanks to its autotranslation btw: I follow a few Chinese-language accounts now that I would have never been able to digest otherwise.

	▲	wbl 9 hours ago \| parent [-]
		Its only high quality of this their thoughts. Autotranslation is different from choosing to have AI shape the expression permanently.

▲

Shadowmist 10 hours ago | parent | prev | next [-]

> Author here.

The best kind of correct.

▲

HarHarVeryFunny 10 hours ago | parent | prev [-]

Why/when would people expect agents to be idling?

I'd have thought they'd be busy (using cached prompt-prefixes) until they were finished.

	▲	what 8 hours ago \| parent [-]
		Maybe waiting on a slow tool call or subagent or something?