Remix.run Logo
▲ zby 2 hours ago

I don't know - all of these cache hit rate advocacy is kind of goodhearting. Cache hit rates are less important than the accuracy of LLM answers and people seem to forget about it. For example I've seen articles how sub-agents poison the cache. Sure you can fill up your context with irrelevant data to have high cache hit rate - but that degrades the LLM work - sub-agents are about providing the LLM with a clean context - just the information needed to do its job, no distraction.

All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.

▲ccgibson 24 minutes ago | parent [-]

I agree with you, accuracy of answers and the ability to complete task is the most important thing. But once a system can achieve a goal, throughput and cost come into consideration. In this regard, I think cache rates help make things both faster and cheaper.