"the intelligence is clearly there"

I wonder if I am using the same models as everyone else. To me, LLMs still give good answers 80% of the time, but 20% it fails in such a miserable way that makes it obvious that the "intelligence" is not there.

I agree. I feel like sonnet 4.6 is sufficient for almost everything. Beyond that level it feels like the orchestration is more important.

That being said the models still surprise me with a broad range of hallucinations, lack of epistemology or common sense or inability to follow instructions on a daily basis.

Today it was trying to get opus 4.8 to just follow a simple architectural pattern for controllers in a rails app. It was pulling teeth out of a shark.

I have both subscriptions and I definitely feel gpt is better and more consistent, but when I run out of limits I don't miss it too much

I am not sure what I am doing wrong then. I am using claude the last 7 months and from time to time try other models like deepseek, kimi etc. Nothing can come even close to it. Claude is almost evrytime (99.99%) one shot.

You might be interested in this:

> With $3.88 & 690,003,591 tokens and 5 hours, Deepseek Pro & Flash combined, managed to reverse engineer Teamspeak's Licensing System for 3.13.8 (latest of post)

https://www.reddit.com/r/DeepSeek/comments/1txcfrh/with_388_...

Can you include GPT 5.5 non-pro (extra high thinking I guess) in your comparison? GPT Pro is the "I am willing to torch cash for a sooometimes slighty better result" option, not the one people are actually expected to use daily. That's probably part of the reason it's not in Codex

Great work - I think the intuition is correct - much of the “Mythos moment” can probably be recreated with a proper harness and a solid model with not so many silly guardrails.

And nice to see the cheap models doing so well.

Where do you run DeepSeek?

I have been saying that from multiple of my tests you can use Claude Code with DS4 Pro or Flash (you just swap api keys) at more or less equivalent performance and people keep screaming "that it's not SOTA".

I don't know whether models are over fitted to benchmarks and people take them at face value, but I spend less on DS4 apis than I do for Claude Code 100$ subscription and I code everyday. So far I'm quite happy with the results.

Depends on what you mean by 'bang for buck'. The open weights aren't better than openai/claude. But they are much cheaper and the limits are much higher, so you get more work out of it for less money. Every subscription provider out there provides better money-per-limit value than Anthropic (other than GitHub, who are by far the most embarrassingly overpriced and limited provider). (https://codeberg.org/mutablecc/calculate-ai-cost/src/branch/...)

> I’m still apprehensive about shipping all my stuff off to a lab under an adversarial government (to the US)

Do you mean you don't want to use the models created by a non-US lab? In that case, yes you're stuck with US models, but there's a half dozen big labs in the US. If you meant just where your inference is done, there are providers in 12 different countries through OpenRouter, including the US. Several subscription providers host in multiple countries. There's a lot of choices.

My advice -- give it a try. Chuck $5 into deepseek.com , and use this config (put it in a shell script, run ' . ./deepseek-claude.sh ', then just run claude as normal.

    export ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic
    export ANTHROPIC_AUTH_TOKEN= *** PUT YOUR DEEPSEEK KEY HERE ***
    export ANTHROPIC_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_OPUS_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_SONNET_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_HAIKU_MODEL=deepseek-v4-flash
    export CLAUDE_CODE_SUBAGENT_MODEL=deepseek-v4-flash
    export CLAUDE_CODE_EFFORT_LEVEL=max

I’m using Claude with a $100/month subscription. I’m playing around with using Opus as the Architect, Sonnet as the implementer/engineer and Deepseek-pro as the deep reviewer, and tester. It’s been quite good as I expected. If my usage pattern holds up, I would downgrade my subscription to the $20/month one and toss more money to Deepseek.

Repo reference here: https://github.com/aravindhsampath/agentic-template

Much more bang per dollar, yes. Somewhat less bang per hour.

As usual, different models get stuck on different things. I run DeepSeek v4 API for most of my Cursor experimentation / poking around / proof of concept stuff, but I trust it less than OpenAI/Claude for writing production code. Sometimes DeepSeek is great for debugging, planning, etc. Sometimes it gets stuck or outputs low quality. That's true of OpenAI and Anthropic models as well though.

Overall, DeepSeek seems serviceable but a rung below Opus 4.8 and GPT 5.5. I run them all on maximum thinking settings.

I've found myself liking opencode for workflows because i can plug GPT models into it, so i tossed 5$ at deepseek api and just toggle back and forth what my opencode.jsonc file is running model wise for my agents. I havent tried anything crazy yet with it, but its nailed all the tasks i felt were overall too simple to waste gpt usage on.

Hardest stuff i threw at it... i did like a set of 3 each for claude/gpt/ds, it was all pretty steady across all providers. I think claude won but it could have just been it rng'd into the 3 easier tasks, they are all similar tasks but not identical, these aren't like benchmark tasks just a steady flow of annoying html/json/regex type stuff. Almost always they need a second pass regardless of what model i throw at it, just to tighten up some loose ends, and it fit right into what my current expectation was of gpt 5.5 and opus 4.6.

I'd recommend carefully looking at a few benchmarks (even though generally relying on benchmarks is problematic)

https://artificialanalysis.ai/evaluations/omniscience

Esp check the Hallucination rate for Deepseek - it's not good.

Yeah, the discounted deepseek inference is subsidized by the CCP for a reason, and it's one that might well come back to bite.

I used ~16,000,000 input tokens yesterday on v4 pro, ~15,000,000 were cache hits, and I spent $0.47. Output tokens were negligible. However that's with Zed's harness, I'm not sure what you would get with Claude Code.

It's maybe not quite as knowledgeable as the most expensive American models and maybe makes more mistakes (just a feeling based off of vibes, don't take my word for it), so you need to constrain its scope more. That suits my workflow, half the time I have it generate code in the chat window and then write it myself, and I'm mostly using it at the level of generating function bodies and stuff, not entire features. Although it is writing a lot of SwiftUI without me really knowing the language and doing a fine job as far as I can tell (which isn't much admittedly).

One benefit I don't see talked about is it's speed - it's really quick, doesn't spend too much time reasoning even on "max", and the flash model is pretty dang good too. This lets me get into "flow state" when I'm writing code, compared to my experiences with Codex and Opus which would take minutes to complete even basic tasks and kind of ruined my focus.

It's so cheap though, you could download a different harness (Crush, OpenCode, Pi etc) and load $5 in credits and test it for yourself.

also curious. On the claude code $200 plan, get close to weekly limits but don't usually hit it. to me just about any small reduction in performance would not be acceptable, the cost of redirecting and getting stuck during long runs without me are too big (like when I tried gemini cli for a few days).

if it's 99.9% comparable performance for less money I'm interested, but I'm skeptical it's there

Have you even used deepseek pro/flash? Yes, it is astroturfed to the maxx. There is a reason for that. The performance/price ratio beats anything available today.

grok-4-1-fast was retired about a month ago.

Requests to grok-4-1-fast-non-reasoning now silently route to grok-4.3 (a 5x more expensive model), with reasoning set to "none".

https://docs.x.ai/developers/migration/may-15-retirement

TFA was published today, which implies grok-4.3 was used.

Pretty small sample size here, but it's hard to avoid the conclusion that DeepSeek and friends will start to put some serious downward pressure on frontier lab token pricing.

Hopefully this dynamic continues long enough to make local/private inference the leading solution for coding.

The OP uses tons of typical AI turns of phrase, and Pangram classified it as AI with high confidence.

So it doesn't surprise me at all that the methodology is weak, too.

That's about what we've seen as well (even directly from deepseek themselves).

We've been using it for async "heartbeat" processing and sms replies, but it's just too slow for live chat replies (which is a shame, as I'd really love to use it there).

Very capable model, but also very slow.

it took me awhile to find a reliable vendor, but they are def out there.

https://artificialanalysis.ai/evaluations/omniscience

They actually explained this a few days back (can't seem to find the link right now). But, the core explanation part was it's architecture.

1. MoE (nothing new here, but, this helps a lot)

2. Compressed Attention Mechanisms (this is their core innovation) - this dramatically reduces the Key-Value (KV) cache requirements for longer contexts

Another thing that helps is significantly lower energy costs in China.

Another point from my own guess: they are running (some percentage) the inference on their own home-grown AI inference chips.

Their models are organized around inference efficiency from the start, it's what they're focusing on. Also they come from HFT and are good at low-level optimization. For v3, they've been literally reverse engineering Nvidia GPUs for undocumented behavior that helped against memory bottlenecks, writing file systems for efficient model serving, and doing a ton of low-level grunt work in the times where everyone else just relied on torch. Being compute-constrained helped as well - necessity is the mother of invention.

That is a very good question. It is open source / open weight - yet none of the third party providers, that also host Deepsek, seem to be able to match Deepseek itself on price.

My guess is that they do aggressive caching / some proprietary optimizations in their hosting setup that they haven't published. Maybe also running at loss to gain market share.

And judging from latency / network performance, I don't think what you access, when you access deepseek.com from Europe, is hosted in China.

I think it's because it's using storytelling-like language to describe reality.

"Harry finally had control of the broom. Draco was dead in his sights. The matchup feels earned."

It's because they assume you know what precision is in regards to this comparison. Normal people don't use such words.

Interesting. Can you elaborate on which harness you've tried it with? I'd love to switch to deepseek for my personal use.

Also, which SOTA western models are you comparing it with? Just to give more flavor.