That table assumes cache hit rate of 95% or better. Am I understanding this correctly that people really are doing such repetitive prompts (compared to each other, across the concurrent user base at that time) that only 5% or less need actually be computed by the intended LLM?

That is shocking. Is it per-token I wonder?

Every tool call is essentially entire prompt so far sent again with the response and that's why cache rates are so high for agentic workloads. This really bites when using expensive models since most models are 1/10 for cached input.

If you are using their coding plan for coding, then yes you can easily hit such cache rates, with a good harness.

I’m getting 97%.