that sounds like a directive that was written to be vague enough to scare any government contractor away from using deepseek for anything ever.

What happens if Linux includes Deepseek code now? All dependencies are poisonous?

Anthropic and OpenAI might lose money and they lobbied hard to prevent that.

For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.

> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload

From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...

Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.

That the model's output is corrupted. See https://aeb.win.tue.nl/linux/hh/thompson/trust.html

"weights open" == "you can download them", not "you know how they landed on them". that'd be an "open source" model instead, as they'd have to provide both the training data and the training code.

hence, the text the model generates is arbitrary. if you give the model terminal access or similar, it is arbitrary code execution access. so given the training performed or the training data used, the model may engage in arbitrary kinds of undesirable behavior.

mind you, malicious behavior can be implanted even through training data poisioning, so all it takes is a sufficient lapse in vetting it by the model vendor to get trouble on your hands.

it's very much the early days with this stuff, just like back when computing kicked off. the big deal was that "thing exists at all". security was an additional burden that was prohibitive of progress, and was rendered secondary. remains true here as well.

I guess you missed this story from last year.

DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.

That's certainly no accident.

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.

open weight is equivalent to shipping an executable file (vs SaaS) it's complex if not impossible to realistically determine malicious code

could you poison the model? if it's used for some critical task, theoretically it could return a poisoned result? not saying it is, just curious

yes, thats the point

These things are true:

- we don’t understand how LLMs work

- DeepSeek is funded through an state-backed investment fund

- China and the US are sort of adversaries

Isn’t that enough to be cautious at the very least?

They're about as indictable as binary blobs found in your BSPs or the 3rd party drivers you get through Windows Update

I don't know if inspection would be practical. How do you inspect the weights for an intentionally hidden trigger that writes malware?

Not only open to inspection, you can also post train it to get exactly what you need from it, both in terms of capabilities and alignment.

Because it is DeepSeek in China, they must have open weight models to be even partially acceptable to the world. I am sure many would balk at using a Chinese company if its products were not open source

It's also odd for them to single out DeepSeek in particular, and not just any model from the PRC?

Most countries permit their governments to access private company infrastructure.

Replace every instance of "PRC" with "U.S.", DeepSeek with OpenAI/Anthropic, and replace "U.S." with any other nation, and maybe remove the last sentence. How does it sound now? Both nations may as well just say it - "we're banning it because they're our closest competitor".

They are also banning local inference: "The models themselves may contain malware or other security vulnerabilities."

It doesn't seem that they have found the models to be actively poisoned, though.

Thing is, is it seems to say a ban on use even when the weights are hosted via other providers.

"third-party products and services developed using DeepSeek implementations"

So you could be hosting it fully locally inside the agency and still it would be banned.

It's not surprising to me, but it's pretty blunt. Not that I care what US agencies do, but this does seem like the first step of many to come.

DeepSeek is just the one with name recognition from the critters in congress. Remember when Nvidia stock had it's so-called "deepseek moment"? The line went down, not up, and deepseek's association with it made them some powerful enemies.

Nobody in congress knows what a "Minimax" is, so they're safe from discrimination, for now.

The others aren't risks to incoming IPO's, yet ;)

They'll go after those too. Give it a minute.

Same reason they attacked Huawei, Tiktok, DJI, BYD... they became too well-known.

What the western elites fear above all else is Chinese brands getting recognizable and popular.

It is THE competitor right now when it comes to price and intelligence. Also, just banning it plants fear in other sponsors and investors. When they do it like this, they can pretend that this is not a part of regulatory capture but it is an actual security concern because "this one in particular is dangerous" and "we'll tell you when others become dangerous" which will happen when they become a competition (now less likely because investors are cautious).

Testing waters, i think

It's not regulatory capture to remove software/harware from adversaries.

No one was saying "Oh McAfee is really gonna control the us's antivirus market regulations" after they banned kaspersky.

That's not why.

We've entered a pretty adversarial era, and we know there are unexpected attack vectors available in frontier models. Just because you an inspect the weights doesn't mean you can derive the behavior.

I suspect the CCP would ban US models if they didn't need to distill them to make their own.

Easily? Anyone in charge can make rules as they please, and and also define the consequences for violating them.

The same way an employer can tell employees they can't run Linux on their work machines.

Open weight.

Anthropic does not allow using their models in China.

Also China bans US technology all the time. Remember the Great Firewall of China? Or when Xi told Trump, “no thanks” to buying NVidia cards