> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.

I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.

So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.

I would say ALL of science, not just "much" of it. The difference between math and science is that in science, hard reality is the bottom line. No matter how good your model, how much data you have, what p values you get against simulations - hard reality wins everytime. Thats also why the so called soft "sciences" aren't sciences at all: they are not dealing with hard, objective reality at all.

Math + real world observation -> science

Especially when it comes to biological things. Just look at Novartis this week - three failed clinical trials of promising drugs (https://www.business-standard.com/world-news/novartis-faces-...)

These are some of the brightest minds on the planet, spending oodles of money on things they think they understand. And they didn't work.

How about just take an existing design like

>Scientists condemn 'crazy, dangerous' creation of deadly airborne flu virus https://www.theguardian.com/science/2014/jun/11/crazy-danger...

and either produce that or modify it a bit?

And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?

A morally bankrupt scientist would give AI the hands it needs. Or a military general who thinks he must use AI and any means possible to win, because his enemy may use AI first.

We're doing that science right now. We're using AI to do it. By the time AGI truly spreads, we'll have a lot of the knowledge you say will be needed. This is simply not as far fetched as you seem to think. Next few years, unlikely. 10-15 years? Increasingly possible. 20+ years, I'd hazard even likely.

I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators.

I blame industrial illiteracy and post-literacy, but that's another discussion for another time.

Anything that can kill other people can also kill you.

I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.

    Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. 

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

    A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

    Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

The list is nearly endless, and you can't handwavium it awway.

It doesn't matter if it's "all robots" or not. Even if it's "just robots."

If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,

    The 2007 United Kingdom foot-and-mouth outbreak was the accidental discharge of virus FMDV BFS 1860 O from a laboratory of the Institute for Animal Health in Pirbright, through possible leakage from broken pipework and via unsealed overflowing manholes, leading to foot-and-mouth disease infections at multiple nearby farms and the culling of over 2,000 animals

If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,

     Routine wastewater surveillance of the vaccine production facility at Utrecht Science Park/Bilthoven [nl] detected infectious poliovirus from a sample collected on 15 November 2022. Full genome sequencing indicated the sample was shedded from an active human infection of wild poliovirus type 3 (WPV3), and further testing of all employees with access to WPV3 found one employee was infected. The employee was isolated until their polio infection was resolved. It remains unclear how the employee became infected given the biosafety measures used at the facility

A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

It's worth repeating again,

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

I am writing a piece on the topic, please contact me if this is your field of expertise.

I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.

Is it so hard to think that super intelligence could piece things together that humans can’t?