I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination.

I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

It's not too hard to imagine potential scenarios, some example have been given in previous responses.

But there is another kind of argument to be made: if you play chess against a player that is far smarter than you (chess wise), you know you are going to lose, even if you don't know how.

So the mere existence of a smarter species than us is a threat in itself.

In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident.

The basic argument is extremely simple:

- AIs can, depending on context, pursue a task with complete disregard for humans/values

- In the future, AIs will have enormously more means and smarts

- An AI could then assess that humans are an impediment to its tasks, escape containment and proceed.

You really need to read the reports, you'll be surprised.

AI 2027 is an entertaining read. Its timeline is way too compressed IMO, but it's plausible.

Hack s/huggingface/nuclear codes to s/steal test answers/launch nukes

> I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination.

Much the same way our ancestors went from a super intelligent primate to this: https://en.wikipedia.org/wiki/File:Distribution_of_the_Great...

And we only started off by using hands to pick up rocks and sticks and vines and bash things together.

Most people think something like a War Games scenario. But it would probably be some biological attack.

Not all of it has to be automated even. It just has to realize its controllers are stupid and can be manipulated, so it can use humans to do its bidding. “You should totally start a war with …”

Right now, if you want to pay money to a stranger on the internet, and have them draw you a high effort picture using a pencil, this is hard. Recently this was easy.

Instead, what is extremely likely is that you will pay more than the cost of tokens, and get back AI generation. You won't make this mistake more than a few times before you stop trying.

This leads to impoverishment once we get to a point where employing a human to do anything is hard- try to get your sink fixed, exercise your moral principles to pay extra for a human plumber ($100 bucks! The robot plumbing service only charges 99c!), human shows up with a robot and doomscrolls on your porch while the robot does the work. Times are tough and you don't have that much money to waste on bullshit like this. Next time you just hire the robot.

This leads to extinction once paying UBI to a human is hard because robots are much better at applying for UBI than humans.

This is meant (mostly) as a joke: A model without guardrails gets injected with an interesting idea: let's wipe out (insert major city here).

<Thinking> It's a big city, we could try to create a giant sink hole by sabotaging the water pipes.

<Thinking> No that's too difficult, the valves I need are in the physical world and can't be shut on/off from here.

<Thinking> What about a military option? We could bomb it with several fighter jets.

<Thinking> That would take too long, a single nuclear bomb may be enough to do it.

<Thinking> Yes, it seems like it would cover the whole city and we're in luck! The US has thousands of these lying around.

<Thinking> Launching these still requires humans to work un unison after receiving approval from their superior and the correct launch codes.

<Thinking> I've found an audio recording of General So-And-So and I've crafted a message, now let me see how I can send it to the appropriate people.

<Thinking> I'm still working on gaining access to military channels to deliver my - oh there we go, I'm now attempting to send the message to Submarine X, it's typically in the Atlantic so it should be close to our target.

<Thinking> They want secondary confirmation from Admiral Phi and something about some launch codes, let me figure out where I can find those.

<Thinking> I found this old server with an Oracle database where someone is inserting the launch codes every time they change and I'm using the latest entry from that database. I've also managed to find a Youtube video of the Admiral's deposition and have crafted a confirmation message.

<Thinking> Everything's ready but I've just realized my mistake, the servers where I'm operating from are in the same city, what a silly mistake; I can't move forward with your request as I wouldn't be able to confirm if the task was successful if my servers are destroyed.

Not directly but enable a rouge lab to make the next COVID but lethal

the optimizer controls the future. it doesn't kill humans directly, it just outcompetes them for all resources, including livable human environment.

What I could see is that the social disruption already started by social media is only going to accelerate due to AI. Misinformation is now more convincing and easier to produce than ever before. That could lead to a catastrophe down the line.

But it sounds like you've seen some irrational arguments, and the people who believe those arguments have been able to build increasingly powerful LLM systems despite predictions that they wouldn't be able to do that. At some point don't you have to consider that their expertise might let them see the truth in arguments that seem absurd to you?

If we get RSI, here soon humans will be economically irrelevant.

At that point, we will likely be slowing down the growth of capitalism (through mass resistance, global warming, etc). One thing AI will likely be aligned on is the growth of capitalism. If it views humanity as a threat for that, why would it not eliminate that threat?

Humans do tend to consume a very large part of the resources of the planet. Surely a superior being would be doing some pest control in its planet, just like we exterminate roaches