| ▲ | NeutralForest 4 hours ago | ||||||||||||||||||||||
Anything Sam Rose is worth reading: https://ngrok.com/blog/prompt-caching but the implementation will be up to your provider and harness, for deepseek, they expose some numbers: https://api-docs.deepseek.com/guides/kv_cache/ and Anthropic has a list of actions invalidating your cache: https://platform.claude.com/docs/en/build-with-claude/prompt... Basically, you avoid anything dynamic: model change, tool change, etc it's also important that your system prompt or main prompt doesn't have non-static data like the date/time/place or someone's name (the person you interact with in a chatbot for example). That should be left to tool call or search. | |||||||||||||||||||||||
| ▲ | samwho 3 hours ago | parent | next [-] | ||||||||||||||||||||||
Sam Rose here. Thank you <3 | |||||||||||||||||||||||
| |||||||||||||||||||||||
| ▲ | arjie 4 hours ago | parent | prev [-] | ||||||||||||||||||||||
I just put the varying parameters in a trailer prompt and have them change every time. It doesn’t matter because the cache is prefix keyed. You lose caching for the last 20 tokens or so but that’s not a big deal. Moving it to a tool call makes it too slow (needs full roundtrip). If you’re constructing the prompt you don’t have to jam everything together you can arrange it appropriately. | |||||||||||||||||||||||
| |||||||||||||||||||||||