The cache shouldn't affect inference. It is purely an I/O optimization.

desterothx 18 hours ago | parent | next [-]

I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper

	▲	dwaltrip 17 hours ago \| parent [-]
		I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.

▲

cma 13 hours ago | parent | prev [-]

[dead]