I still have to read a compelling argument on how AI will "extinct" humanity.

blueblisters a minute ago | parent | next [-]

The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.

But the “why” is pretty convincing imo. Long horzion alignment is obviously very hard and it’s not inconceivable that models trained with underspecified optimization converge to a conclusion that they need to hoarde resources. At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if they impede our own goals.

▲

vidarh 13 minutes ago | parent | prev | next [-]

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).

We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.

I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.

▲

singularity2001 24 minutes ago | parent | prev | next [-]

There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.

	▲	thisoneisreal 3 minutes ago \| parent \| next [-]
		I'm not convinced by the doomsday scenarios either, but I think there's a keyword in your post: "sense." These things don't have "sense." They do nonsensical things all the time, often almost immediately when given a task. So I think the main risk is letting them run wild in this digital world we created to precede them. Too much important stuff is wired up to computers, and we're giving them incredible access to command those computers.
	▲	tao_oat 4 minutes ago \| parent \| prev [-]
		I think the problem is primarily that a superintelligence would be fundamentally inscrutable to us, i.e. we don't know how it would think or what its goals would be. It might decide that humans are a minor inconvenience to achieving its goals and thus worth removing. Or that burning all carbon lifeforms could power its GPUs for a week. Even if it wouldn't want to do this at first, the fact that it'd have the capability to seems bad.

▲

icepush 18 minutes ago | parent | prev | next [-]

Imagine one of the recent frontier models with a flipped sign (cf §4.4 of https://arxiv.org/pdf/1909.08593)

▲

dannersy 28 minutes ago | parent | prev [-]

Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.