> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.

This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?

If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.

▲

intorio an hour ago | parent | next [-]

> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.

Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?

	▲	fidotron an hour ago \| parent [-]
		Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot). The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology. There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.

▲

alach11 2 hours ago | parent | prev | next [-]

I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!

Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.

	▲	fidotron 2 hours ago \| parent [-]
		> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file! So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?

▲

OkayPhysicist 2 hours ago | parent | prev [-]

You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.

You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.

Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.