| ▲ | dools 3 hours ago | |
Orly? “ Automatic Prefix Caching (APC in short) caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part.” https://docs.vllm.ai/en/latest/features/automatic_prefix_cac... | ||
| ▲ | wahnfrieden an hour ago | parent [-] | |
This thread is about cutting costs in half for GPT across the board. The technique you linked only makes a substantial difference for particular use cases where you are going to have many LONG CONTEXT queries with the same prefix. For instance, when having a set of documents that commonly get loaded in as context. It's a way for application developers to keep prefixes they manage (or prefixes managed by some set of their users) cached. It has no relevance for long tail general purpose use. | ||