I find this - or perhaps the title - a bit surprising.

I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.

Perhaps DS works better where targets have low-hanging fruits than can be found fast?

▲

mariopt 2 hours ago | parent | next [-]

Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.

Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.

▲

seemaze 2 hours ago | parent [-]

How does GLM 5.3 Flash rank against it's big brother and the latest Deepseek?

	▲	networked an hour ago \| parent \| next [-]
		Keep in mind that GLM-5.3 isn't GLM-5.3-Flash's "big brother" the way one might think. GLM-5.3-Flash is not GLM-5.3 scaled down. While GLM-5.3 is based on GLM-5.2, and "every gain comes from post-training" (https://z.ai/blog/glm-5.3), GLM-5.3-Flash uses a newly trained multimodal base model (https://z.ai/blog/glm-5.3-flash).
	▲	mariopt an hour ago \| parent \| prev [-]
		Both these models are not bad but you can not trust them to make good decisions. I meant GLM 5.1/5.2/5.3 in the previous answer, not the flash version

▲

Aissen an hour ago | parent | prev | next [-]

If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.

	▲	an hour ago \| parent \| next [-]
		[deleted]
	▲	networked an hour ago \| parent \| prev \| next [-]
		> If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless. Inference providers can credibly promise to not train on your data if they are in a position to get sued.
	▲	an hour ago \| parent \| prev [-]
		[deleted]

▲

simlevesque 2 hours ago | parent | prev | next [-]

In your example, couldn't you parallelize DS's work more ? You could have 11 times as many agents for the same price.

	▲	TuxSH 2 hours ago \| parent \| next [-]
		Both DS and GLM had the same numbers of subagents, 5 or so. But, well, number of subagents doesn't make a difference if model is dumb (GPT 5.4 High, in May,in Chat mode outperformed what I see with DS4.1-F). That being said, pricing model makes a huge difference for "find at least one" tasks: with API/PAYG if you have a chance to save 90%, you go for it, whereas with subscriptions it is optimal to burn all your remaining allowance right before reset
	▲	d5lt5 2 hours ago \| parent \| prev [-]
		DS has perf issues if you parallelize it heavily (24+), especially when the context window is above the limit, on a single machine (with custom llm gateway): it fails 4x more often, and is 2x slower than gpt-5.6.

▲

severino 2 hours ago | parent | prev | next [-]

A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.

	▲	drdebug 30 minutes ago \| parent \| next [-]
		You can run deepseek-v4.1-flash on a local server with 4xH200@141GB for around 180K EUR. It can serve more than one user so it can be a good investment for a company, especially if you need your data to stay local (although I would advise do go a step up and populate with 8xH200 to serve many more users, ~300K EUR). You can run whatever test you want and you won't be limited by a front-end that pre-filters your requests.
	▲	SSLy an hour ago \| parent \| prev \| next [-]
		GLM-5.3; well, the flash variant at least; you just ask what to do and it complies with no objections :^]
	▲	TuxSH an hour ago \| parent \| prev [-]
		GLM 5.3 doesn't seem to refuse vuln research (which it classifies as "audit") and is good at it. Therefore you use for offensive cybersecurity tasks because Daybreak Red/Mythos is pure unobtainium for us mere plebians. DB Blue thankfully exists, but I suspect you risk a ban if you use it with codebases you neither own nor use Tl;dr because it's the only model at the level of 5.4~5.6 that doesn't refuse tasks nor risk your oai account getting banned For plain RE tasks Sol or Astra should work just fine (I think)

▲

allie1 2 hours ago | parent | prev | next [-]

I think it really depends on what the data DS was fine tuned on. If your use case is very specific, it wouldn’t have distilled that knowledge well.

▲

himata4113 an hour ago | parent | prev | next [-]

you can run 10 ds-f agents at the same time for the same cost exploring a much wider surface area.

▲

surgical_fire 2 hours ago | parent | prev | next [-]

What I find surprising is that DS Flash can do it at all.

I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.

▲

cmrdporcupine 2 hours ago | parent | prev [-]

Thing is that GLM 5.3 is many multiples the cost to run, and slower.

I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.

So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.