Isn't caching a server-side thing? How does the agent affect it, significantly at least?

embedding-shape 6 hours ago | parent [-]

Say you put the current time down to the second in the system prompt, which is the message that goes in front of the entire conversation, then basically nothing will be cached, every agent turn needs to ingest the entire session over and over. Contrast to not doing that, and the backend can leverage caching all the way up to the latest message, as nothing until then changed.

▲

esperent 6 hours ago | parent | next [-]

Surely other agent CLIs are not dumb enough to invalidate cache on every turn over something so obvious?

	▲	chillfox 5 hours ago \| parent \| next [-]
		I don't think any the agents breaks caching on every turn, but they might do things like current list of files, or available tools depending upon plan/build mode... or lots of other things that breaks caching multiple times during a session.
	▲	brookst 5 hours ago \| parent \| prev \| next [-]
		Probably not that exactly, but there is a tradeoff between effectiveness of the prompt and cache hit rate. If putting the user’s datetime in the middle of the prompt scores higher on evals but worsens cache hits, versus at the end of the prompt where it’s cache friendly but may not be as effective, what do you do? This is still art as much as science and the different harnesses take different approaches.
	▲	embedding-shape 5 hours ago \| parent \| prev [-]
		Obviously not, most agents properly keep previous messages unchanged, at least the major ones I've been digging into the source off. Also, everything would get so much slower, that even developers creating their own agents would notice quickly how much slower theirs is, if they fuck this up.

▲

theanonymousone 5 hours ago | parent | prev [-]

Yes, of course you can destroy it. But how far can you "improve", beyond decent "common sense" behaviour.