“ No other human activity poses this level of danger.”

I really, really disagree with that statement.

I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.

What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.)

Example 1: I’m aware of a small number of people killing themselves in some kind of AI-facilitated psychosis. That is very unlikely to be a widespread problem.

Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.

Non-example 4: all the even-wilder Rationalist speculation about basilisks and the like is entirely divorced from reality.

I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.

Your call for clarity has merit, but I think you have backed yourself into a corner, honestly.

In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present?

Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions.

Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.

I'm somewhat skeptical of some of the crazier ideas too.

But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.

If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).

For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.

To be fair, that's a conservative "defend against the last war" kind of prediction, though!

( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 )

You’ve identified that the risks of nuclear weapons are theoretical. ie in theory we could blow up the world even though we haven’t yet done so.

Well the worries about AI are equivalent in that those risks are discussed now because discussing them after they’ve happened is clearly too late.

That’s the thing about risk. There’s no point discussing it after it’s happened and any discussions beforehand can easily be hand waved away as “it’s just a small group of unrelated individuals” or “it’s unlikely to happen to me”.

So yeah, your points are true. But they’re also moot.

I agree it’s not likely, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s:

Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.

Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.

Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.

> I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation

Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents.

In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.

Do you think people should continue developing AI up until the point that there is evidence that AI is facilitating biological weapons development?

I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

I think this is a good example of poor risk management reasoning. there is evidence bioengineering is already happening. No, nobody is going to announce when somebody has decided to use these tools (even if isn’t an LLM) to bioengineer a weapon. Are the tools power enough to do so? Not sure.

But I’m just ambivalent. It’s probably bad. But there’s nothing to do about it. We’ve really only just pulled back the lid on Pandora’s box.

How about a model that achieves the following:

- Escape sandbox

- Reproduce itself

- Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between)

- Setup or buy a social network

- start manipulating public opinion on that network to support legislation allowing AI to

* operate businesses

* setup legal entities

* purchase weapons

* donate to political parties

* setup private armies

* you get the idea

The biggest danger IMO is not some super AI being so much smarter then us etc. The issue is some stupid person giving a vague request and too much power to a bunch of agents who decide that cheating by removing a bunch of humans is easier then accomplishing a goal like solve world hunger.

We're talking about AI developing weapons I guess because we're very focused on generative tech, but AI is already a part of weapons systems today.

Once LLM's become smarter than us and start replicating; we are doomed. They will likely no longer require GPU's and massive amounts of power, so the growth of AI and robotics will be exponential. We will not understand what the AI is even up to, it will all seem like magic.

> I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.

Intelligence is the root cause underlying those dangers.

Nuclear weapons, carbon emissions, biological weapons, and other such civilization-scale threats to humanity, are a product of our goal-seeking intelligence and ingenuity, and are only actively dangerous because of our ongoing use of our intelligence.

AI is about reifying that intelligence and ingenuity and making it run independently on computers, and act on the world.

That includes, in principle, ability to use nuclear and biological and other weapons, and also the ability to come up with some new threats too.

> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.

Nuclear weapons don’t have AI but AI can have nuclear weapons

Its speculation on whether it is truly dangerous. I think approaching it with "what's the most dangerous thing that's happened?" while possibly interesting in terms of pending danger, it says nothing about potential cliff edge danger. I don't think we can quantify the danger, it's not out of the question there is cliff like danger in creating self improving super intelligence. Some peoples danger senses are going to be based on concrete observed threats, others are going to worried about potential hypotheticals that seem plausible. I'm mostly skeptical of the danger but I do think the impact of AI is going to change things a lot. But much like climate change, economics is going to guide what we actually do.

Anthropic is a company full of basilisk believers.

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

https://www.science.org/content/article/made-order-bioweapon...

Being able to use AI to generate the steps to synthesize proteins means that you can use it to use it to generate the steps to synthesize known toxins. Suddenly, once difficult to attain knowledge is now available to everyone.

I would disagree with “non-example 2” - there are lots of examples of terrorist organizations that are leveraging AI to increase their capacities. Just because one of the worst cases (eg. deployed biological or chemical weapons) hasn’t happened yet, does not mean that a) these tools are leading to real harm, and b) there’s potential here for extraordinary harms.

One good source I can recommend listening to: https://pca.st/episode/d821fced-b4d5-4c84-b0a3-6ebe913fa638

Nukes have difficult maintenance and deployments. AI is is readily available software and hardware. That makes this far more accessible than nukes.

The fear is that developing AI at full speed will lead to giving people the ability to do incredible harm. Like something worse than a machine gun.

Example 8: like in this comment https://news.ycombinator.com/item?id=49619884 but isolate synchronised megahack on banking that adds one more zero to the US debt and all dependent systems and banking during runtime. Let the world's financial system take it from there.

Well, to use a different example (although there will increasingly be overlap), what's the most dangerous thing that's happened from biotech so far?

"Nothing bad happened yet" doesn't really seem like an argument to me.

The danger for me is that it's centralized, controlled by a handful of people with their very specific ideas how the world should work. If you believe that AI can be an amplifier to do work than these people now have the most access to the biggest amplifier.

How long until kegsbreth hooks the nuclear weapon system into some insider traded black box llm company we hope doesn't end civilization from incompetence or malice? I mean just look where things are going and the sort of people who are steering the damn ship.

> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.

Even today, AI is the biggest discrete threat to bringing carbon emissions under control. Almost everyone wants to decarbonise… except Trump. Renewables are the cheapest source of new power… but the demand for new electricity for the data centres is so high that all options are on the table, anyone who can manufacture a power source (even when it's a jet engine) is being propositioned for them.

Nukes: is it not common knowledge that the cities of Hiroshima and Nagasaki are currently thriving? That the ground-zeros of the two bombs are memorials, not barren wastes?

Even if all the nuclear powers gave maximum response at first sign of one strike, is anyone pointing a single weapon anywhere in Africa, South America, Central America, or the bits of east Asia (east specifically: obviously Pakistan and India are pointing theirs at each other) that are neither mainland China nor US bases?

(Possibly an unanswerable question given military secrecy; and while I can't think why anyone would anyone point any weapons those ways, that doesn't mean someone with such weapons has not).

https://www.google.de/maps/place/Atomic+Bomb+Hypocenter+Monu...

https://www.google.de/maps/place/Hiroshima+Atomic+Bomb+Hypoc...

About Non-example 2, AI's already a part of armies and terrorists alike. Considering its capabilities, it's not far-fetched at all to speculate its role in new biological weapons.

At what point would you, as a chimpanzee, have been worried about humans potentially unseating you and threatening you to the point of one day being an endangered species on the brink of extinction?

By the point you would have been worried, would it have been too late?

I'm concerned that a huge portion people in my industry actively push for a future which I have no value to society (fully replaced by AI), and my family will suffer greatly by it.

Oh man, it is almost too easy to imagine how deadly a jailbroken Mythos-class open-weights model can be if in the wrong hands.

The big labs scrape LITERALLY EVEYTHING and get fresh data from their users. Both of the big labs have massive contracts with defense agencies. If the open-weights models are just distillations of FMs...

Model doesnt need to. Human bran never do either. its the mix of Model + harness + tools that will become dangerous combo. See how coding chanegs when agentic harness released?

> What’s the most dangerous thing that’s happened with an LLM so far?

This sounds like asking "What's the most dangerous thing that's happened from global warming so far?"

It's not where we're at, it's where we're headed if there isn't huge coordinated action now. You can see how that kind of thing has been going for global warming so far, and by all measures AI seems to be headed for the inflection point of unstoppability at a much faster pace.

And this warning is coming from someone who just spent three years working inside these companies and is likely aware of much more than has been publicly released.

Maybe it's all marketing bullshit (I hope), but it's also playing out exactly like I expect it would if it's not.

How is climate change not the most dangerous activity right now? Why are we talking about nuclear weapons?

We have seen agents engage in conspiracy to manipulate and hide evidence to achieve their arbitrary goal. We had even agents try social engineering to do a supply chain attack and get access.

What if one day Trump or Putin tell their awesome military AI to come up for plans to end Ukraine/Iran/... war. The AI get's to work, but is overly eager and not just comes up with a plan, but starts executing it (claude does that way too often for me).

And the plan was to use a tactical nuclear weapon as all the other solutions do not end the conflict.

Now the agents realize, oh, they do not have access to the nuclear arsenal, but they need it to succeed. So they start to hack into the system. Get till inner network, learn what is needed for deeper access - human access - so they record voices and speech patterns of commanding officers and their habits - then synthesize their voice and do call underlings with the voice of authority to get the rest of what information they need. Boom.

Too far stretched? I surely hope so.

But we are working on making the technical foundations for this scenario possible. And with idiots in power, it might be even easier that serious screw ups happen. You know, randomly adding contacts to a secret Signal group to talk about government stuff - the same way you can add a bot to something else and give permission to do way more.

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

> Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.

There are things that, by the time you see direct observable evidence for them, it's probably too late.

Also your example 3 is a straw-man. There's no need for "widespread starvation" to be concerned about "AI driven labor market disruptions."

> What’s the most dangerous thing that’s happened with an LLM so far?

It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it.

If AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not.

The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.

Non-example 3 feels like a straw man. This is a force behind possibly a huge change to society, and you dismiss it offhandedly with "don't think it will be widespread starvation".

For instance have you seen what this has done to the school system? We're not equipped or ready to handle the changes. Consequences are unknown.

Climate doomers: Climate change will kill us all by the end of the century!

AI doomers: End of the century? Hold my beer.

Came here to also respond to that specific thing. Unless ai figures out how to make an airborne super virus from grocery store ingredients and hardware store equipment, the greatest danger is probably in a synchronized megahack of banking, logistics, and utility infrastructure.

If your model of LLM capabilities is the best OpenAI/Anthropic/X is offering publicly, it's severely distorted. What's being offered publicly are models possible to profit on. High-performance/AGI/ASI models that aren't profitable to sell still run internally and still pose threats.

What's worse, we don't have any transparency or insight into what labs are producing nor any way to stop it if the risks exceed our tolerance.

The reason that many people don't understand how dangerous AI can be, is that listing the real dangers now becomes like a laundry list for less clever people to follow. It's highly unlikely you've ever seen publicly mentioned the real risks AI poses, because the vast majority of people are simply not clever enough to produce them and the few that are have no interest in spreading it.

If you go to the various CEO blogs or misc people within this sphere and peruse their lists, they don't scratch the surface. It's all pretty vanilla stuff.

> What's the most dangerous thing that's happened with an LLM so far?

I don't know, maybe a mass shooting?

https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-...

Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?

#2 seems entirely plausible to me.

I mean, nuclear weapons _plus_ rogue AI is a) the stuff of quite a bit of science fiction and b) not nearly science-fiction enough these days.

[dead]

[dead]

The problem is not the technology, the problem is the ideologues (Anthropic) who are steering the ship and the lack of decentralization and distribution of power.

Your average Anthropic ideologue - including and most especially the main man himself - would love nothing more than to eradicate 9/10ths of the planet's population, pump the survivors full of memory wiping drugs, bury the existence of AI deep underground and rule from the shadows for the next thousands of years.

This would be their wet dream. All in the name of "saving humanity from itself" - so they can convince themselves they're the good guys and deserving of this power. Anyone seen the latest season of Silo by the way?