Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).

> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.

This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?

If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.

> ...I see how no-guardrails LLMs could be truly disastrous...

by all accounts this is what's what because of how...

> You can do CRISPR in your kitchen.

.. and because of the same dynamic we see occurring in cybersec.

If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.

If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H

Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.

If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.

This experiment has been run, and even the biological stuff is far fetched.

I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.

The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.

For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.

After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.

Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.

This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.

Just because you know something about a thing doesn't mean that you're capable of doing that thing.

Let's take bioweapons.

They keep comparing wet work in a lab to writing code on a computer.

When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.

You can screw up an infinite number of times on your way to a successful exploit.

If you screw up with lethal agents in a lab? You die.

Here's a non-exhaustive list,

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

   A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

   Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

   Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.

The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.

Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?

No, it doesn't work like that.

The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.

And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.

These experiments have been run. And we can run them again.

The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.

Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.

Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.

What if "way short of AGI" is still dangerous in ways we can't easily handle?

i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.

we're always in uncharted territory. that's why most HN commenters think that this is hyperbole.

i'm not trying to be rude, but i really don't get what you mean by this? (https://www.amazon.com/Superintelligence-Dangers-Strategies-...) there's been people making specific and unique risks to this technologies for years, before AI was even present. anthropic was literally founded because they were worried about existential risk? also, previous technologies have had existential risks? do you deny that nuclear weapons could have killed humanity? obviously, there have no been existential risks that have killed all of humanity, because by definition, that can only happen once. it will literally be impossible for someone to ever point to an example of that happening, because they will be dead by the time that happens.

See printing press and internet!