datsci_est_2015 7 hours ago

Digression, but this is the real Great Filter imo, not AI. I think technology advances to a point where it only takes one or two bad actors to type the right prompt to get a recipe for civilization-destroying bioweapons before you get anywhere near true AGI or anything relevant to the Kardashev scale. Biology is fragile.

But not that that’s a good justification for hamstrung models. I think it’s just the inevitable endgame and it’s more sad than scary

▲

jeremyjh 7 hours ago | parent | next [-]

I have the same concern. If it becomes possible to engineer Captain Trips with a budget in the low 8 digits it won’t really matter what else happens.

▲

xyzzy123 6 hours ago | parent | prev [-]

I don't fully understand the instinct to regulate local models for this? It seems like the wrong place to address the problem.

You can download Ebola sequences right now if you want to. That's not the same as having an isolate. The difference is a lot of messy reality. This kind of work is not generally "one shot" (Claude make me a supervirus, make no mistakes), it requires lab space, iteration, and specific resources. It has a footprint.

Wouldn't it make more sense to monitor / regulate facilities where you can sequence or request assembly of DNA, RNA, restrict and monitor the supply of key reagents and so on?

▲

datsci_est_2015 3 hours ago | parent | next [-]

Yeah I was thinking more on the scale of nation-state level actors. Most leaders would not consider a bio weapon due to potential blowback on their own population, as they generally don’t want to be the leader of a nation of corpses and ashes.

But if they don’t care, or if they already are the leader of a nation of corpses and ashes, maybe there’s not so much holding them back.

In my layman’s perspective, bio weapons seem to be more dangerous and volatile than nuclear weapons. And increasingly easier to engineer as biotech advances. I think about the headlines of “microscopic robots that target tumors from within your body” and see the horrifying opposite side of the double-edged sword.

▲

addaon 5 hours ago | parent | prev [-]

It depends how easy it is (now, or in the near future) to turn information into weapons, and how realistic control of materials is. There's a reason we control access to plutonium, but information about metastable hafnium.

	▲	xyzzy123 4 hours ago \| parent [-]
		It does seem to me that for this specific problem the materials are a lot more amenable to control than the information is? There's also this weird revealed threat model thing going on? Like why does it make sense to support heavy LLM restrictions but leave benchtop oligo synthesisers completely unregulated? (Note: I do agree that wanting to regulate BOTH is at least a consistent and defensible position). I find it philosophically interesting because the problem is not strictly information control. Local models don't have any special extra information with respect to biological research. What has to be restricted is using information that's already publicly known in the wrong ways.