| ▲ | danpalmer 6 hours ago | |
The kv cache is not persistent. It's a hyper-short-term memory. | ||
| ▲ | in-silico 4 hours ago | parent [-] | |
Modern kv caches can contain up to 1 million tokens (~3000 pages of text). It's not that short, it's like 48 straight hours of reading. | ||