is a kv cache not a kind of state? what does statefulness have to do with selfhood? how does a system prompt work at all if these things have no reference to themselves?

▲

danpalmer 8 hours ago | parent [-]

The kv cache is not persistent. It's a hyper-short-term memory.

	▲	in-silico 6 hours ago \| parent [-]
		Modern kv caches can contain up to 1 million tokens (~3000 pages of text). It's not that short, it's like 48 straight hours of reading.