Remix.run Logo
▲ torben-friis 3 hours ago

Can you explain how this removes their moat? Genuine question, I'm not sure I understand enough about LLMs to get how an open source inference engine undermines them.

▲mbesto a minute ago | parent | next [-]

When the model is fungible and the switching cost is near zero (business logic lies in the harness), then there really isn't a moat for models.

▲frays 3 hours ago | parent | prev | next [-]

Today the closed-source giants maintain a substantial cost-per-token advantage due to proprietary inference tricks like speculative decoding kernels, ultra-optimized FlashAttention derivatives, multi-GPU pipeline orchestration and dynamic prefix-caching algorithms.

I'm not sure whether the DeepSeek's open-sourced inference engine does all of these, but it appears that plenty of inference tricks have just been revealed for the open source community to learn from and start improving upon.

▲torben-friis 3 hours ago | parent | next [-]

So... this makes it easier for someone in the position to run chatgpt class models as a service to get similar returns to openAI/anthropic, if I got you.

And I imagine this engine is by default mostly just deepseek compatible and inference engines are not universally applicable to all models?

It sounds significant for competition but I'm not sure how it sinks Anthropic/openAI. Presumably they still have their frontier models, the training data they're vacuuming from current users and access to hardware to serve at a ridiculous scale, right?

▲schneehertz 3 hours ago | parent | next [-]

DeepSeek can accomplish over 90% of tasks of equivalent value at one-tenth of the fully loaded cost of O/A, and this capability can further spread through open source. In my view, the IPO risk for O/A is no longer just a valuation discount, but a substantive challenge to its long-term business model as a high-margin foundation model provider.

In that scenario, even if global Token usage increases 100-fold in the future, that alone would not be enough to justify these foundation model companies being worth one or two trillion dollars.

▲mobelkh an hour ago | parent | prev [-]

I think for me it's more the possibility of the ideas here being incorporated in other Opensource projects (vllm, llama.cpp, or in the architecture choices of the models themselves)

Deepseek manages to be ridiculously cheap and efficient with prompt caching, i spent 550M~ (1.9b input cache miss, 1.7b output) tokens and it cost me 5~ bucks, that would be ~150 bucks on Opus 5.5, if i ran that workload in parallel that would be $50 vs $1500, e.i. i wouldn't want to subagent heavily with anthropic models.

now imagine Kimi K4 or GLM 5.5 with those cost savings

▲asdf88990 3 hours ago | parent | prev | next [-]

I think you’re correct, although suggesting that open AI is a trickle drown of AI companies is a bit naive.

But the key point is that something good enough always attracts more interest and contributions because a productive outcome is very realistic and tangible.

Open Weight models and competing open source inferences engines massively undermine monopolies; in theory at least.

See how cloud is easy to divest and “nationalise”.

▲FeepingCreature 2 hours ago | parent | prev | next [-]

All of those are in open-source engines. I'm not convinced they have that much of an advantage.

▲CurbStomper4 3 hours ago | parent | prev [-]

[dead]

▲trentor 3 hours ago | parent | prev | next [-]

What moat?

▲kasey_junk 2 hours ago | parent | next [-]

It is, and always has been, data contracts. Which is why the hyperscalars are a more likely candidate to take American frontier business than Chinese vendors.

I can click a button in Azure foundry and run open weight models that are approved through my compliance and procurement teams.

I don’t know how open sourcing this stuff helps deepseek but it means I’m getting ever better and cheaper models so I’m fine with it.

▲torben-friis 3 hours ago | parent | prev | next [-]

Well, whatever real or perceived moat Openai/anthropic had. OP claims this open source inference engine reduces their chances of high valuation, I'm assuming this is due to their in house engine being something they could use as differentiation before?

▲mrweasel 3 hours ago | parent | prev | next [-]

That is probably what has been the most fascinating part of this AI/LLM boom, the moat was apparently never the technology. It has been very much like nuclear energi (and weapons), once you know how to do it, it's not that difficult. It's still fairly hard, but you can do it. The moat has been the funding for the infrastructure and if that dries up you're in trouble.

▲wg0 3 hours ago | parent | prev [-]

Yeah what moat is the right question to ask.

▲dividedbyzero 2 hours ago | parent [-]

I've been trying to find an answer to that, too. The best I've got right now is this:

Moat-like things going for them:

- access to the US government (for huge inflated government contracts as a source of free capital, having the competition banned, having their products legally mandated, getting other preferential treatment, having foreign governments bullied into granting similar advantages)

Too unsustainable, too fragile, too small:

- Strong brands: Brand loyalty seems like a pretty insignificant factor in AI right now

- Semi-exclusive hardware access, RAM buyouts: Unsustainable even in the short term

- Lock-in effects: Their models aren't very stable across versions, or very unique, and switching providers is about as easy as it can be in a highly competitive market.

- Exclusive technology that is important, advanced and hard to replicate to a degree that would materially set them apart and justify potentially much higher cost. Maybe I'm missing something here, but a lot of the claims they make to that end fall apart even on light scrutiny (due to things like test sets that are small, hand-picked and clearly overfitted on, or broad general claims that cannot be substantiated). I'm getting the feeling they don't have anything that materially sets them apart at all, but that's seems impossible to measure in any meaningful way, so who knows, they might have something here.

▲throwa356262 3 hours ago | parent | prev [-]

In theory, this can allow a small US lab to compete with O/A/G and take marketshare from them with small open models deployed centrally or (and this is think is the real moat) on-prem for enterprises.