I asked a family member biologist about concerns over open weight models helping to create bioweapons:

I can imagine a situation where a bio weapon pathogen could potentially be engineered to evade the immune system by creating a pathogen that is both extremely virulent and has a rapid mutation rate. Rapid mutation rates make it difficult to create an effective vaccine and also difficult for the immune system to learn to recognize the pathogen.

But whether open weight AI models can allow people to create such bioweapons - it does seem far-fetched to me. I think it's far-fetched because biological systems contain so many elements that we don't even know about, do not exist in any data set, and so they could not be included in the model.

In order for a pathogen to be successful, there are a number of environmental barriers that it has to endure in order to reach the host, survive in the host, proliferate in the host, and damage the host. Environmental barriers outside the host and inside the host.

I know that AI can be used to design genes to encode synthetic proteins. So could an open weight AI model result in the creation of genes that encode harmful proteins and then bioengineer pathogen to produce the proteins. But there are so many unknowns that can affect whether this pathogen can survive at all. So I would say from my limited knowledge, I would say it's far-fetched.

You hope it’s “far-fetched”.

As noted in this paper⁽¹⁾, ”Publicly releasing model weights is an act of irreversible proliferation. Once weights are released, they can never be recalled.”

Do you really want to risk it on your off-the-cuff reasoning that it’s “far-fetched”?

⁽¹⁾ https://www.lesswrong.com/posts/qmQFHCgCyEEjuy5a7/lora-fine-...

I'm sure you have thought the same thing about CRISPR and cloud labs in the past. But this is real and will probably doom us all.