Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it.

We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point.

For example you never want your system prompt to start with the time that the session started. Move that to the first user message if needed.

▲

macNchz 4 hours ago | parent | next [-]

Caching is not supposed to work like that, but that doesn’t preclude the cache key computation function from having bugs.

	▲	marginalia_nu 4 hours ago \| parent \| next [-]
		Yeah there's quite a lot of potential bugs that could have this shape. If I were to guess it could be a buffer in a buffer pool not being sized and zeroed correctly, allowing stale data to bleed between sessions.
	▲	nok22kon 2 hours ago \| parent \| prev [-]
		or the cache retrieval function for a key retrieving the wrong entry

▲

Waterluvian 3 hours ago | parent | prev | next [-]

There is a massive incentive for optimization, so I expect they’re doing a ton of very clever tricks, all of which make this kind of bug more likely.

▲

estebarb 3 hours ago | parent | prev | next [-]

Hash functions necesarily have collisions. Also, it is perfectly possible to introduce bugs in the hash function (hash inputs, hash function itself) that allows cross account contamination.

	▲	margalabargala 2 hours ago \| parent [-]
		Hash functions necessarily have collisions, but it's perfectly possible to make the expected time between collisions greater than the human lifespan.

▲

supriyo-biswas 4 hours ago | parent | prev [-]

There could just also be a bug where the output tokens of session 1 were shared with session 2, due to a race condition or similar.