To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?

The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.

The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.

The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.

Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.

> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?

In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.

But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.

If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.

"Power corrupts and absolute power corrupts absolutely." Lord Acton

I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.

Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.

The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.

If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.

> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?

Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!

The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.

But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.

If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.

the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"

There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.

> what should be done about open weight bioweapon and cyber-offense capabilities?

Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.

> Is it simply the cost of freedom that we should allow attackers to access these tools?

Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.

> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.

I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.

This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.

I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:

https://news.ycombinator.com/item?id=49078376

----

And related thoughts on past posts:

https://news.ycombinator.com/item?id=49034988

https://news.ycombinator.com/item?id=48516722

> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?

Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.

> Is it simply the cost of freedom that we should allow attackers to access these tools?

Bad actors WILL have access. The question is will these mega corps stop innovation?

There's a difference between:

> Something should be done

and

> Something can be done

In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.

The US can attempt to stop those models from being trained in the first place but good luck with that.

The difference with open weight is that everyone has access to the same weaponry

The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.

maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust

>To everyone here pushing for total proliferation of ...

...general-purpose computers

...unbreakable encryption

...unbackdoored communication

...unkillswitched vehicles

...unsurveiled dwellings

>what should be done about ...?

nothing

>Do you seriously want this level of capabilities to be generally available with no guardrails?

yes

> what should be done about open weight bioweapon

Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )

> and cyber-offense capabilities?

You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.

The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.

> what should be done about open weight bioweapon

The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.

In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.

Everything you said could apply to computers many decades ago. Think of the nuclear fission simulations our enemies could carry out!

We'll be fine.

The software industry should be ashamed by the number of exploits that ai can find in software. It’s really an embarrassment.

The software has to be built better.

There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.

I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.

The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.

[dead]

> what should be done about open weight bioweapon and cyber-offense capabilities?

If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.