| ▲ | _aavaa_ 3 hours ago |
| Notably missing their kv cache system |
|
| ▲ | freakynit 3 hours ago | parent | next [-] |
| It's exceptionally optimized... 890 bytes per token. At standard 32K context, that's less than 30MB. 1. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash 2. https://www.mindstudio.ai/blog/deepseek-v4-1-flash-specs-arc... |
| |
| ▲ | entrope 3 hours ago | parent | next [-] | | Sure, but that size/token is inherent to the model. Part of DeepSeek's hosting advantage, and thus a remaining part of their moat, is migrating KV cache from VRAM to host RAM to nonvolatile storage, and presumably routing requests to where KV cache has migrated. People here on HN say KV cache lifetimes can be 8+ hours with DeepSeek, and I don't know of any other provider that can match that. That infrastructure isn't part of this open-source release. | | | |
| ▲ | _aavaa_ 3 hours ago | parent | prev [-] | | Yes, the theoretical approach is listed there. But the actual code is what's missing. |
|
|
| ▲ | GreenWatermelon 3 hours ago | parent | prev [-] |
| yeah they mention: > Required for a complete inference service (not included in this release): > A KV cache system > VE/Prefill/Decode disaggregation (most required engine functionality is already implemented) > Load balancing across data-parallel replicas for both prefill and decode |
| |
| ▲ | _aavaa_ 2 hours ago | parent [-] | | Yes, that's why I commented. I'm not sure I understand your point. | | |
| ▲ | GreenWatermelon 2 hours ago | parent [-] | | I was just adding a citation to your point, not arguing against you :) also adding what other stuff aren't included in the release, and implicitly pointing out that they explicitly mention it's not included rather than it being stealthily missing. |
|
|
