| |
| ▲ | minraws an hour ago | parent | next [-] | | As someone who recently tried it on some blackwell cards, it's possible to match the prices especially the input can be even cheaper and output can match the costs so you can easily build a net 20-30% margin business even at current GPU prices. The entire issue is caching, I tried to write some custom to dump to disk kv-caching using some ideas from their papers and my experience with snapshots and vm checkpoint systems, I must say they must have really squeezed that lemon it's hard. Atleast me with Sol couldn't figure it out over a couple days, a few hours each day, which isn't much but I did feel a bit stuck with existing solutions and felt like I might have to write something from scratch. But if you are willing to put in the effort into the infra I do think it's doable. But it will be really hard to pull it off. My congrats to anyone who manages to pull it off, they might be able to kill off most AI labs. Assuming they can find the compute, Deepseek really has killed all models for me other than Sol/Fable/Opus/K3 tier stuff. | |
| ▲ | twotwotwo an hour ago | parent | prev | next [-] | | One read is 1) they're getting a lot of traffic for Flash, 2) they've said they're updating Pro soon and expect that to lead to a traffic spike for Pro, but 3) that would leave them overloaded, so 4) they're going to raise prices to avoid it. It's interesting that most open models adding 1M context did it in a way that reduces KV cache size (though DeepSeek was the most aggressive, using compressed attention on all layers), but only a couple providers turned it into a discount on cache reads. | |
| ▲ | _aavaa_ 19 minutes ago | parent | prev | next [-] | | I'll believe it when I see it. Their prices are still much higher than deepseek, especially the caching. | |
| ▲ | NorwegianDude an hour ago | parent | prev | next [-] | | Eh, what are you guys even talking about? Deepseek is not cheapest provider as is, and it's MIT. So deepseek making it more expensive to use is just nonsense, they can only change their own pricing. It's the beauty of MIT license and open weights. If anything, these models are some of the safest in the world to use if you worry about a rug pull. | | |
| ▲ | akman an hour ago | parent | next [-] | | 90%+ cache hit rate is common, and so you'll see on places like openrouter that Deepseek cache cost is indeed a magnitude cheaper than the rest. | | |
| ▲ | greenavocado 11 minutes ago | parent [-] | | My usage thus far from api.deepseek.com - input_cache_hit_tokens: 1,265,646,976 x 0.0000000028 = $3.5438115328
- input_cache_miss_tokens: 18,208,088 x 0.00000014 = $2.54913232
- output_tokens: 9,615,178 x 0.00000028 = $2.69224984
- request_count: 10,837 (no price)
Total cost: $8.7851936928 (approximately $8.79)Cache: - Hit: 1,265,646,976
- Miss: 18,208,088
- Total input tokens: 1,283,855,064
Hit rate: 98.582% (1,265,646,976 / 1,283,855,064) |
| |
| ▲ | LaurensBER an hour ago | parent | prev | next [-] | | There's more to inference than just the input/output token cost. Caching has a massive impact. Deepseek charges $0.0028 per cache read on Openrouter. The next cheapest is $0.018. That's a massive difference and quickly adds up on coding sessions (which often hit 95%+ cached tokens). | |
| ▲ | hagen8 34 minutes ago | parent | prev [-] | | Cached input tokens are what drives most costs. |
| |
| ▲ | retinaros an hour ago | parent | prev | next [-] | | any link to this caching tech? | | |
| ▲ | LaurensBER an hour ago | parent [-] | | [Feat][Core] Add disk offloading support to SimpleCPUOffloadConnector — #49644
https://github.com/vllm-project/vllm/pull/49644 This adds disk as a tier in the HBM → CPU → Disk KV cache hierarchy. There's also a cluster of related KV-offload FS PRs: #49225 (read/write batching, still open) and #49152 (batch store/load in C, merged Jul 28). It's hard to say if these are similar to the approach DeepSeek takes but they definitely seem very interesting. |
| |
| ▲ | onlyrealcuzzo an hour ago | parent | prev [-] | | > Deepseeks secret sauce is the incredibly cheap caching (magnitude cheaper than other providers). Can anyone working at one of the main US labs (Google, OpenAI, Anthropic) comment on WTF they haven't even tried MLA - despite the obvious massive advantages? I know enough to know they aren't completely incompetent. So there must be a quite good reason. But it remains a mystery to me. DeepSeek's MLA is like almost 2 years old at this time. They've got thousands of people working on this stuff. They clearly have the ability to at least try it... | | |
| ▲ | aabdi 20 minutes ago | parent | next [-] | | They already are? There’s a measurable performance tradeoff versus gqa so there’s reluctance. For the most part though the new deepseek v4 tech is hca and mhc and people are still catching on like with moe and rl. Wait for 6 12 months, minimum time for next pre train. | |
| ▲ | ronsor an hour ago | parent | prev [-] | | Are they not? The big US labs are opaque and don't publish much of any technical details anymore. We don't know what they are or aren't doing, honestly. |
|
|
