Remix.run Logo
AI Can Now Design Functional Viruses. Should We Worry?(spectrum.ieee.org)
46 points by pseudolus an hour ago | 60 comments
SubiculumCode 17 minutes ago | parent | next [-]

AI is an existential threat, full stop. It can also help us solve many of our greatest problems. Let's not let it solve those problems by removing us as the quickest shortcut.

hoppp 31 minutes ago | parent | prev | next [-]

The regulation at the DNA Printer level works I think, as long as nobody invents a good at home DNA printer, but if they do that there is no need for custom viruses, somebody can take an existing one that is known to be deadly and just print it.

dotancohen 9 minutes ago | parent | next [-]

Aren't technological constraints such as these short lived? Even if s home DNA printer won't be available this decade, the technology will come at some point. It's far better to have both legal and social policy in place before then - though the legal policy will probably not matter at all.

HarHarVeryFunny 6 minutes ago | parent | prev | next [-]

A custom designed virus is potentially far more deadly than something like ebola, because animals/plants will have zero resistance to it.

It'd be like releasing an invasive species into an ecosystem that has not co-evolved to co-exist with it.

idontwantthis 28 minutes ago | parent | prev [-]

That’s what I’m thinking. If we can restrict EUV technology to the West then why can’t we restrict DNA manufacturing? Not to the West in this case but to capable governments like with Nuclear power.

ben_w 15 minutes ago | parent | next [-]

An EUV factory is a moderately sized building in its own right[0]; on the other hand, DNA is manufactured separately inside most of the cells in your body[1], the plant on your windowsill, the mould on your bread…

[0] "43 freight containers and weighing in at 165 tons." - https://www.theregister.com/offbeat/2024/12/03/asml-releases...

[1] not in red blood cells

estearum 20 minutes ago | parent | prev | next [-]

My understanding is DNA synthesis is now like a 2nd or 3rd year undergrad tier challenge and doesn't even require much specialized equipment.

Versus EUV which one could dump literally unlimited amounts of money into solving and still probably fail for decades.

morkalork 14 minutes ago | parent [-]

I want to say I've already read some articles about hobbyists buying previous generation equipment from labs. The entire supplychain would need to be locked down with used equipment being returned to manufacturers for there to be any security.

sscaryterry 5 minutes ago | parent [-]

I saw a sweet electron microscope on eBay, for 2000 quid. If I had the space...

convolvatron 26 minutes ago | parent | prev | next [-]

its not really a question of will, its a question of difficulty. if it were easy and cheap to build EUV processes, no amount of complaining and posturing would prevent someone from reproducing that.

Onavo 19 minutes ago | parent | prev [-]

Not sure if you realize it but biology is a lot harder to regulate. Most of the secret sauce in biotech is in the manufacturing process, not the reagents itself.

I will give you a classic example, the Lipid Nanoparticle used for the delivery of mRNA vaccine is the real lynchpin of covid vaccines. The media never talks about it. Any biology student knows how lipid bilayers work (they are the basic building block of cells, it's like asking a programmer what a terminal is). But the process of manufacturing those lipid nano particles at scale was the main bottleneck for delivering mRNA to the target cell. That was the trade secret. It's not sexy or hot like mRNA so the media doesn't talk about it. It's why many developing countries tried to make their own vaccines and failed completely. The mRNA was the easy part but what use is a bunch of mRNA if you can't get them delivered to cells?

Any biology student (even high school ones) can describe how a cell membrane works. But designing a man-made one that can be created at scale and be guaranteed to fuse with mammalian cells is mostly a matter of trial and error. It's a bit like finding viable pseudo linear memory use alternatives to the basic attention architecture in machine learning. Every lab has their own take on it.

With AI nowadays the search space of viable biotech ex nihilo experimentation is much smaller given that you can just prompt it to give you plausible protocols.

timr an hour ago | parent | prev | next [-]

The design was never the hard part. You still have to make the thing.

Designing a modifcation to an existing virus is perhaps more achievable, but the implementation is still the hard part.

haxiomic 39 minutes ago | parent | next [-]

There’s YouTubers who do this [0]. I’d argue estimate the risks of a synthetically induced pandemic of massive impact this century are very high

- [0] https://youtu.be/aoczYXJeMY4?is=OqObeGzZB6AKL0Jk

timr 4 minutes ago | parent | next [-]

Not even close to the same thing. That's taking an existing viral transfection system and swapping in a single gene.

Thermo Fisher sells a kit to do this.

wat10000 18 minutes ago | parent | prev [-]

“There is another theory which states that this has already happened.”

But the next one (or the first one, if that one wasn’t manmade) will probably be much worse.

estearum 44 minutes ago | parent | prev [-]

You can literally send an email to one of hundreds of companies that do exactly this. They have no way of knowing whether you're asking them to synthesize a virus or a drug candidate or what.

timr 22 minutes ago | parent | next [-]

No, you literally cannot. Synthesizing a stretch of DNA is trivial, and what you can send away for [1].

Building a functional virus is not even remotely a turn-key operation. Assuming you could get the genome, expressing that genome in a way that results in infectious virus is non-trivial.

[1] Even then, the companies will rarely give you long enough reads to make a full genome scaffold. It looks like there are two companies that will do 50-100kb assemblies as a service today, but I guarantee you they're going to look at what you're trying to make, and who is asking them to do it.

estearum 15 minutes ago | parent | next [-]

https://www.genewiz.com/public/services/gene-synthesis/custo...

There you go!

Steps from your mailbox to an actual virus is a few days of bench work.

timr 7 minutes ago | parent [-]

Yes, that's one of the ones I was talking about.

That's not a service that makes you a virus, and making one from an mRNA transcript is not trivial.

rbliss 21 minutes ago | parent | prev [-]

Yet.

timr 8 minutes ago | parent [-]

No, the sophomoric LLM argument does not apply here. The world is not a big computer.

The point is that the rate-limiting step in producing something infective is a couple million dollars in special lab equipment, skilled practitioners, and incredibly tricky and tedious procedures that take time and repetition to master.

So yeah, you can try to pay someone who has that expertise to make your top-sekret superbug (good luck with that), or you can do what any real threat actor is doing to do, and put in the time and investment. That's the hard part, not asking ChatGPT for a genome sequence.

Let me put it another way: despite the synthesis pathways for many common illegal drugs being widely known and trivial enough that an undergrad O-chem major could do them, in fact doing this is a very good way to get caught and arrested. And that's easy.

estearum 3 minutes ago | parent [-]

Your example of how easy it is to control outputs behind specialized knowledge is a thing (recreational drugs) that exist in mass quantities every place on the planet at consumer-accessible prices?

demibabs 30 minutes ago | parent | prev | next [-]

Would they not be like “hmmm this one looks kinda similar to a known virus”

They really just synthesize anything for you?

estearum 28 minutes ago | parent | next [-]

I'm sure some of them would, but they're not required by law to do so. And historically the way you'd do this is by actual fragment/substring matches. But these are novel viruses now. So what you really want to do is try to predict whether a novel genome is pathogenic... which is itself obviously a very hard technical problem.

Enginerrrd 5 minutes ago | parent [-]

That’s not too much a concern I think. Most of the real pathogenic qualities result from known genes. Inventing a totally novel virus that is also pathogenic is basically science fiction right now. Splicing together known components that could be bad in combination is closer to current technology.

smallmancontrov 18 minutes ago | parent | prev [-]

No, of course not. In 2010 I went to a guest lecture from a guy who worked on the scanning algorithm for IDT. A few years later, there was a minor scandal when someone in a nearby lab used a personal credit card for a purchase and got a big shared account locked. So not only were they scanning the content of synthesis orders, they were validating payment back to labs. In 2010, when HTTPS was rare and facebook didn't even offer it.

estearum 14 minutes ago | parent [-]

"There are websites where you can just buy prescription drugs online?"

You: "No, of course not. I once went on a website to buy a drug, and they made go to a doctor and get a prescription and send it to them"

smallmancontrov 7 minutes ago | parent [-]

The legit synthesis sites have been scanning and authorizing since before the internet was encrypted. You suggested otherwise, demibabs expressed surprise, and was correct to express surprise because your implication was incorrect.

The black market exists -- but comes with black market problems.

transcriptase 40 minutes ago | parent | prev | next [-]

Yes in much the same way you can literally build a rocket and go to the moon.

You may be skipping a few of the involved steps.

estearum 36 minutes ago | parent [-]

What involved steps are those?

You email the RNA strand (a text file) to a lab along with some money. The lab converts that information into a physical object (read: virus) and sends it back to you in the mail.

What step is missing?

smallmancontrov 15 minutes ago | parent [-]

The part where that service doesn't exist and isn't close to existing.

estearum 13 minutes ago | parent [-]

https://www.genewiz.com/public/services/gene-synthesis/custo...

Distance from mailbox to live virus is a few days of bench work from a semi-competent biochemist.

And yes, this service claims to have safeguards. Does every service? Does every future analogous service? Are those safeguards guaranteed to be robust against totally novel pathogens?

Answers: No, no, and no.

smallmancontrov 2 minutes ago | parent [-]

Really? mRNA? You really don't know anything about these protocols do you?

consensus1 38 minutes ago | parent | prev [-]

One of the easiest and most effective AI regulations would be to simply require all these labs to run all incoming requests through a suite of the most advanced models asking if the request is likely to be a pathogenic agent.

estearum 34 minutes ago | parent | next [-]

Yes probably true. Unfortunately we need only one defector lab to severely degrade the overall effectiveness of this approach (leaving aside the effectiveness of the detection models themselves).

wat10000 20 minutes ago | parent | prev | next [-]

Who’s going to simply require that? There are ~200 sovereign nations on the planet, they’re not all going to do it, or have effective enforcement if they do.

ben_w 7 minutes ago | parent [-]

This is the kind of thing that would be relatively easy to convince nations to sign up to*; actually enforcing, I suspect you're right.

* there's not an obvious benefit to being the one nation that says "you know what? We want to be known as 'the nation whose biolab insecurity wrecked the world', because taking blame works out so well for everyone.", though I'm fairly sure a large portion of the world will be totally willing to wargame the impact of hosting a lab as a deadman switch so nobody else attacks them first…

dgellow 26 minutes ago | parent | prev [-]

That’s one interesting way to generate demand for AI vendors if the AI bet doesn’t pay off /s

evilelectron 13 minutes ago | parent | prev | next [-]

Get ready for the next adventure in the saga of 12 Monkeys, with AI starts like Bruce Wills, Madeleine Stowe, Brad Pitt and Christopher Plummer, coming to a bio lab around you. The experience of a life time one you can't miss, it is in the air.

acjohnson55 37 minutes ago | parent | prev | next [-]

Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical free market) encourage reckless aggressiveness at the level of companies and countries.

Although, I must admit, as a doomer, I'm pleasantly surprised things aren't already worse than they are.

energy123 13 minutes ago | parent | next [-]

This is what AI safety activist Connor Leahy calls offense-defense asymmetry. Imagine if a nuke could be created with household materials. We got lucky that wasn't the case, but we didn't know that it wasn't the case before studying physics. If we weren't so lucky that physics happened to turn out that way, we already wouldn't exist, because the offense would be so much stronger than defense, we wouldn't stand a chance even with the best surveillance systems.

A profitable prediction markets trader who goes by the alias semiotic rivalry posted this prediction on his X account recently:

> Prediction: in 1-2 years, almost everyone will agree that open sourcing the best AIs is totally crazy as the AIs become capable of autonomous hacking of most humans and institutions, serious terrorism and creation of biological weapons, etc.

And I mostly agree with him. As much as it's awful to have such a concentration of capabilities in the hands of a few governments or corporations, as far as I can tell there are no easy answers here, only a matter of trade offs and different risks.

andsoitis 36 minutes ago | parent | prev | next [-]

> proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people

did you mean greater number of people with less resources?

EDIT: thanks to others that point out how you can interpret this differently, which I think is: "it is now easier for smaller teams with fewer resources" rather than my original interpretation of "power is now concentrated so that fewer people can do this than before"

gtowey 33 minutes ago | parent | next [-]

No, I think they mean that it used to take a giant lab full of specialists with lots of funding to do lots of damage before.

Now, the worry is a lone nut in a basement with no money could pull off an act of bio terrorism.

estearum 33 minutes ago | parent | prev | next [-]

I had the same reaction, but I think they meant fewer as in "requires less consensus-building to wield meaningful power." Used to have to convince lots of people to go to war in order to kill a couple million people. Fast approaching a one- or two-man weekend project.

demibabs 28 minutes ago | parent | prev [-]

Fewer as in the size of an individual group, even though the total number of people who could find themselves in such a group is greater.

thatfunkymunki 16 minutes ago | parent | prev | next [-]

I think you would get a kick out of reading this paper on this exact topic, called the Vulnerable World Hypothesis https://nickbostrom.com/papers/vulnerable.pdf

However I think the result is disappointing, it concludes the only solution is a massive preventative surveillance state and global governance.

wat10000 12 minutes ago | parent | prev | next [-]

I’d say that is almost the definition of technology: that which allows doing ever more with fewer and fewer people and resources.

And technology doesn’t really care what you use it for. It can be constructive or destructive.

It “only” enhanced local destructive potential, until last century. Then it got to the point where a huge effort could produce a truly global catastrophe. And then a bit smaller effort. And technology continues to march on.

Eddy_Viscosity2 31 minutes ago | parent | prev [-]

Every person's utopia is someone else's dystopia and vice versa. What you've described is clearly some peoples (ie. already billionaires) version of the best possible world. Not so much for the rest of us though...

bmacho 8 minutes ago | parent [-]

I'm so surprised that HN is this much anti-government. I'm a doomer myself, I fear the rich, I fear the terrorists, and I think the only one that

  - can save my life, 
  - is interested in saving my life
  - i have some control over
is the state.

Yes, they are currently snatching kids of (illegal) immigrant parents and it is horrible, but who else would protect me against lone nuts in a basement, or robor armies of trillionairs?

mehraban 28 minutes ago | parent | prev | next [-]

Yes, we should, but even here distribution is an serious issue.

clickety_clack 26 minutes ago | parent | prev | next [-]

Were we safer when the viruses were object oriented?

ModernMech 23 minutes ago | parent | prev | next [-]

Doubly because they’ll also use AI to manage the AI pandemic response.

wild_pointer an hour ago | parent | prev | next [-]

Betteridge's law strikes again

falcor84 41 minutes ago | parent [-]

Wait, why are you not worried?

estearum 25 minutes ago | parent [-]

It's super cringe and d e c e l to think technological advancement could have any dangers whatsoever.

Just like in a car, the true demonstration of intelligence is closing your eyes and slamming the pedal to the floor.

Alive-in-2025 3 minutes ago | parent [-]

You are describing some people driving with FSD today. I bet hundreds or maybe thousands of people a day try letting it drive for a while without paying any attention. The car watches your eyes to be sure you don't do this, but people try to defeat that too. Sunglasses make it hard to tell what's going on.

I see it's still happening as of July 2026 -

https://www.techtimes.com/articles/319863/20260707/sunglasse...

https://www.reddit.com/r/TeslaLounge/comments/zt1kgt/anyone_...

But even though there are a few people tricking the "are you paying attention" check, probably more people complain about it being too careful and overbearing.

colingauvin an hour ago | parent | prev [-]

No, a human could do (and have, many many many times, see like half of molecular biology) this as well. If anything this will help with vaccine development.

Edit: I'm a PhD biochemist that has done significant work on viruses. Love the downvotes. Never change HN. ¯\_(ツ)_/¯

nozzlegear 6 minutes ago | parent | next [-]

I agree with you. I favor the view that humans are generally morally good when taken on the whole. There already exists the technology for ne'er-do-wells to engineer viruses and unleash them on the world. But I submit that A) there aren't many people who want to do that in the first place; and B) there's a vanishingly small amount of people who want to work with those people to help them achieve it.

estearum 39 minutes ago | parent | prev | next [-]

Same reason it couldn't be a problem to sell aerosolized anthrax at the hardware store. Humans have (many many times) made aerosolized anthrax. Can't see any conceivable problem with more people having access to it.

dgellow 23 minutes ago | parent | prev [-]

That doesn’t invalidate the point, you’re just saying it’s already a risk