If you read DeepSeek's papers, you'll find a litany of architectural features that allow for a greatly reduced cache hit price by shrinking the size of the KV-cache.

▲

yfontana 6 hours ago | parent [-]

How come no other big model seems to be able to deliver the same type of extremely low cache cost though, if their techniques are public?

▲

jboss10 5 hours ago | parent | next [-]

I think the "architectural features" are part of the model, not the kv cache. So implementing it would be difficult and expensive.

▲

petu 5 hours ago | parent | prev | next [-]

Deepseek V4 paper is just ~three months old

▲

sudosysgen 5 hours ago | parent | prev [-]

Many of these techniques haven't been published very long ago - it often takes a good 6-8 months for techniques to percolate. But also, they come at a complexity cost and, seemingly, also at a stability cost.

	▲	hnfong 2 hours ago \| parent [-]
		Also potentially a performance (in terms of output quality) cost. DeepSeek is cheap on a per token basis but lags behind in the benchmarks, perhaps it was a calculated tradeoff.