Remix.run Logo
icepush 3 hours ago

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

CalRobert an hour ago | parent | next [-]

Ocean deoxygenation comes to mind.

thomascountz 3 hours ago | parent | prev | next [-]

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

computerphage 3 hours ago | parent | next [-]

Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

BartjeD 12 minutes ago | parent | next [-]

Global warming causing war between nuclear armed opponents. Migration flows and competition over ever shrinking resources.

It's silly to treat these things as separate categories as if only one will happen at a time. These issues are happening, and are going to happen all at once. That is the fundamental challenge. AI plays into that because it can make wars and nuclear exchanges so much more effective.

If an AI tells the President the USA can survive China's nuclear barrage mostly unschathed, perhaps he'll press the button...

mrpopo 3 hours ago | parent | prev | next [-]

Runaway warming leading to a hot "supergreenhouse" climate was theorized. Alterbatively a snowpiercer scenario creating a snowball earth.

Or, once billions of people die from climate change, planetary wars will start on the last hospitable areas, leading to the end of humanity.

In contrast, AI ending "humanity" is limited by our own material power, would likely be stoppable, and would not easily reach areas isolated from technology

pingou 2 hours ago | parent [-]

Why would it not easily reach areas isolated from technology? If a nefarious AI wins a war, it would be one of the easiest thing for it to locate any survivors anywhere on the planet.

ben_w 2 hours ago | parent [-]

I suspect you two are discussing different AI.

I think you (and I, and everyone who wants AI development to pause while we catch up with the implications at least) is looking where the ball is going.

I have the general (non specific to anyone in this thread) impression that people who think it can't end humanity, are looking where the ball is today.

Current AI obviously can't "locate any survivors anywhere on the planet".

Where the ball is going… well, much as I don't believe Musk's timelines for anything, he is trying to sell his Optimus robots as a "robot army", about them running factories, about factories on the moon etc.

I suspect the moon-factory "idea" was someone asking Grok, given how the numbers don't really work for building compute as well as power, but the scale of that is enough to change Earth's equilibrium temperature by… I forget, but IIRC it's many tens of Kelvin rather than single-K from global warming if this was all put in LEO for some reason.

thomascountz 2 hours ago | parent | prev | next [-]

There's a lot to unpack with this, and I am not an expert, but: crop failure across industrial monocrops leads to food shortages, wild fires destroy significant areas which support pollinator ecologies, fresh water become contaminated by rising sea water and blighted by drought, and perhaps most significantly, as resources become strained, humans become desperate and armed conflict proliferates. This is just top of mind.

ben_w 2 hours ago | parent [-]

None of those are "all"/extinction.

Bad, sure; I wish they were not present to threaten us, but they are not close to "all".

Humans are extremely resilient omnivores. We are able to eat a huge range of things from algae to zebra, and get food (farmed and hunted) from salt water as well as fresh.

thomascountz 2 hours ago | parent [-]

Perhaps I have a bit more imagination—or a bit more humility towards our humanity—such that I believe this is not "either/or". In the vast search space of potential future outcomes, of course even I can see humanity survive nature as I've seen nature survive humanity. However, I can also so easily see probable branches where we are our own undoing. Apropos of TFA, if one is to believe LLMs can carve a threatening path to humanity end, how could one not see the same for our much older extant threats?

ben_w 27 minutes ago | parent [-]

> Apropos of TFA, if one is to believe LLMs can carve a threatening path to humanity end, how could one not see the same for our much older extant threats?

Which "much older" extant threats?

Global warming and nukes are industrial era threats. Pre-industrial threats were the four horsemen of the apocalypse: war, famine, pestilence, and death.

War and death didn't go away, but are not (and never were) extinction threats. Famine and pestilence, well, there's a reason people point the finger of blame at the Chinese government for the Great Leap Forward, and why we had lockdowns for Covid while we tested that the vaccines were safe (we went from "genome sequenced" to "first human test" in 66 days; the rest of the wait being mix of "yes but how sure are we it's safe?" tests and manufacturing at scale).

If everywhere except North Sentinel Island was wiped out, or everywhere but Hawaii, or everywhere but Greenland, humanity would be back to the level of the 1750s within a few millennia at most; and other than the North Sentinel Island example, likely even faster as books with many advanced solutions to historical problems would survive and be comprehensible.

And the problem with LLMs (and other AI) is, we're giving them control of stuff, and that stuff includes robots. The *current* versions mainly concern me for economic and cybersecurity reasons, but the tech moves fast, and the companies working on them are reckless.

My core expectation is these models will cause 1e3-1e7 deaths in a single event before people actually take the risk seriously. The lower end of that is "industrial accident", the upper range includes "convinces people to go to war", and these are examples of ways current LLMs can already plausibly do wrong, given who uses them, why they use them, and how careful they are about their usage.

However, a common failure mode for people is "that was an unfortunate incident, but we studied the problem and we fixed it, it can't happen again" right before a different error they hadn't though of explodes in their faces. This is how we can get to 8e9 deaths: people keep pushing the models to do more, and keep lying to themselves that "this time we fixed all the problems, everything will work, it will be great".

mitthrowaway2 2 hours ago | parent | prev | next [-]

Worst case is warming oceans create a hypoxic environment where anaerobic bacteria thrive en masse and generate H2S over country-scale areas undersea. The chemocline breaches the surface and the ocean and atmosphere become poisonous to most complex life and agriculture, also stripping the ozone layer in the process, irradiating the surface. This is one mechanism posited for the end-permian mass extinction that eradicated most ocean and surface life, including the trilobites.

Not sure how long we'd survive such a scenario, even sheltering underground. But surely it couldn't happen to us.

(However, it now seems like the AI might get us first.)

3 hours ago | parent | prev [-]
[deleted]
icepush 3 hours ago | parent | prev | next [-]

Global warming is happening at a slow enough pace that there is plenty of time for people to move to the poles/into caves/beneath the ocean/whatever. Nuclear war would be much more destructive in a much shorter range of time but you will have pockets.

Findecanor 27 minutes ago | parent | next [-]

People already entrenched near the poles will fight to keep others out. Right-wing anti-immigrant parties are on the rise right now already.

They also tend to not want to recognise climate change -- because that would imply that people could migrate because of it.

thomascountz 2 hours ago | parent | prev | next [-]

Citation needed for evidence of human cooperation at this scale. As you say, climate change is slow moving, and there's no real evidence across the last 60 years of knowing about it suggesting humans are willing to do what it takes to survive it.

ben_w 2 hours ago | parent | next [-]

All human behaviour since no later than the Polynesians turned the Pacific islands into a network of sailing trade routes.

"Humans move" is one of the easiest things to predict. The biggest problem today is that they generally move to somewhere that other humans are already in, which is of course rather lessened by any major disaster wiping out a majority of the population.

icepush 2 hours ago | parent | prev [-]

What cooperation are you referring to?

thomascountz 2 hours ago | parent [-]

   ...people to move to the poles/into caves/beneath the ocean/whatever
Cooperation required for the purported mass migration of civilization. Admittedly, I assumed you meant it would be civil, but perhaps that should be excluded from the priors.
icepush an hour ago | parent [-]

I never made any predictions of a cooperative (Civil or otherwise) mass migration of civilization - just that a nonzero number of people would move from areas that were becoming uninhabitable to areas that were becoming habitable. Did you reply to the right comment?

toilet 2 hours ago | parent | prev [-]

Climate change isn't real because we can just move under the oceans hahahaha

icepush 2 hours ago | parent [-]

If you read my comment very carefully, you will find that is not exactly what I claimed

pingou 3 hours ago | parent | prev | next [-]

How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that.

They could destroy most civilisations, culture and scientific achievements though.

nurumaik 2 hours ago | parent | next [-]

Because nuclear war balances itself: more nuclear bombings = less nuclear weapons, less people to send them. It will naturally stop at some fraction of humanity left which is incapable of making and launching more nukes

Dylan16807 2 hours ago | parent [-]

The general idea of a nuclear apocalypse is that all the nukes that matter are launched in a matter of hours, and anyone being targeted gets their launches out before they're hit. The feedback loop that slows things down is too late to matter.

King-Aaron 3 hours ago | parent | prev | next [-]

In the exact same way major sudden changes in the climate have lead to the extinction of the majority of plants an animals over this planets history. We are not special.

pingou 3 hours ago | parent | next [-]

We are special in many ways, we have technology, we are everywhere, and we can think and plan (although considering global warming that is debatable). Global warming would have to create dramatic conditions everywhere on the planet, not leaving any small pocket of survivability to make humans extinct.

Possible? Theoretically yes, but pretty unlikely. But obviously extreme global warming would be a catastrophe even if some of humanity survive.

IsTom 2 hours ago | parent | next [-]

Technology depends on a lot of people cooperating around the globe. Disrupt a few critical chains (power generation, fertilizers, computers), add a bit of good old war and you'll soon be in the era before Haber process and a lot of people die. And then the remaining people will have trouble keeping that level of technology with how sudden the change would be.

pingou 2 hours ago | parent [-]

Well yes that's what I said, it could destroy civilisations and their technology. It seems much harder to kill every isolated tribe in the whole world, and every survivors of the destroyed civilisations.

thomascountz 2 hours ago | parent | prev | next [-]

There's little evidence that humans are able or willing to cooperate at this scale. See: the past century of climate change.

jddj 2 hours ago | parent | prev [-]

Runaway greenhouse effects on other planets have resulted in surface temps of 400C+

ben_w 2 hours ago | parent | next [-]

Yes, but Venus has 93 times the atmospheric pressure of Earth, and it's overwhelmingly CO2.

Earth isn't likely to get a true runaway like that until the sun gets another billion or so years on the clock, and when it does it will be water vapour as the oceans are promoted to atmosphere.

What we're doing to ourselves is still bad, of course, but it's nowhere near that bad.

pingou 2 hours ago | parent | prev [-]

Is there a possible scenario for that on earth, that results in every place on earth having a temperature non survivable by humans?

Dylan16807 2 hours ago | parent | prev [-]

We have a massive survival range and we're on every continent. That kind of sudden climate change is not nearly enough to wipe us out. Some other climate scenarios might. Nuclear war definitely could.

King-Aaron 2 hours ago | parent [-]

The thing that would end our species due to nuclear war is the exact same thing that would end us from climate change - a collapse of the modern systems of society that we depend on for survival. A nuclear exchange won't set every human on fire, but it's the collapse of food production, healthcare, logistics chains, security, and eventual disease that follows. And because of the size of our population and how depended the majority of us are on these systems, it would likely happen extremely quickly before plateauing out to very small, scattered population groups that would struggle in a hostile environment.

Dylan16807 2 hours ago | parent | next [-]

Outside of extreme feedback loop scenarios, there is no way for climate change to disrupt food anywhere near the level of nuclear winter, and there would be no mass destruction of supply chains either.

King-Aaron an hour ago | parent [-]

Welp extreme feedback loops are indeed on the menu.

s1artibartfast an hour ago | parent | prev [-]

So you agree they would survive

King-Aaron an hour ago | parent [-]

Functionally extinct. I would wager that isolated tribes clinging for survival in a post-climate-disaster world may not progress very far, but that's kind of a moot point to argue over.

Balinares 2 hours ago | parent | prev [-]

Humans need food. Food does not just appear in supermarkets.

ben_w 2 hours ago | parent [-]

While true, the impact from global warming is likely to be less crops rather than no crops.

s1artibartfast 2 hours ago | parent | prev | next [-]

I would much rather take my chances with either over AI. Both at once.

xienze an hour ago | parent | prev [-]

Humans are voluntarily driving themselves to extinction through sub-replacement birth rates worldwide. That's going to play out far quicker than global warming or pretty much anything other than every country on earth launching nukes at each other will.

kakacik 33 minutes ago | parent [-]

Hard disagree here, population growth is still happening and will keep happening for some time, maybe step out of your bubble if you don't see that.

Africa is still growing massively for example, world is not just western civilization. Sure at that rate and incerase of living conditions for everybody maybe in 1000 years population will be smaller, but its not that hard to fix if wanted - people used to have 10-15 kids as default.

elAhmo 3 hours ago | parent | prev | next [-]

Neither does an LLM model.

frabcus an hour ago | parent | next [-]

Oh indeed, but an LLM model that's reasonably smart could brute force find other training algorithms that are that dangerous. Just as it solves maths problems.

This is the explicit plan of OpenAI and Anthropic.

ben_w an hour ago | parent | prev [-]

LLMs can already control robots. Today's LLMs are inept at it, but they can do it.

Things will be bad, unless someone does something to stop it. Will anyone do that? We don't know.

https://xkcd.com/2278/

XorNot 2 hours ago | parent | prev [-]

Arguing whether "all" or just "most" of humans dying is certainly a choice.

Survival of the species isn't enough.

yoble an hour ago | parent [-]

In prioritizing risks I do feel it's a genuine qualitative difference though.

Number of humans going to 1 million would be huge catastrophe, but after a few thousand years it gets back up to billions of people, renewed every generation.

Number of humans going to 0 means that's it. One scenario has many orders of magnitude more missing humans, when you count future lives.