I heard the following analogy which made a lot of sense to me: suppose you're playing a chess match against Stockfish. Stockfish will win. Even if I cannot tell you what moves it will play, I can tell you with certainty how it will end.

Similarly, we cannot predict what AI would do.

▲

danbruc 2 hours ago | parent [-]

This assumes you are not trying to prevent Stockfish from wining. I can do many things from using chess engines myself to just smashing the computer that can or will lead to other outcomes than Stockfish beating me.

▲

johnthewise 2 hours ago | parent | next [-]

Yes, stockfish is confined to moves within a chess game so you can stop playing the game.

Can we say the same thing about the agents? Current, probably. But doesnt it look like everyone is spending all their effort integrating&connecting them everywhere, so they are not confined & do more on behalf of us?

It's easy to imagine a scenario where we would just shut down a very intelligent agent cluster. Is it hard to imagine though the same agent can have also access to that to prevent us from doing it? this defense would be more plausible if we weren't racing to give them every tool&act.

▲

number6 2 hours ago | parent | prev [-]

And in terms of AGI it is, that the AI can't survive without humans, and if we decide to quit the game than its over for the AGI; we will happily regress in a techno-barbarian feudal state and salvage solar panels and trade copper wires while still reproducing and carring on. We are playing a whole different game here.

▲

johnthewise an hour ago | parent | next [-]

If AI is threatening enough that we can collectively just decide to stop it, wouldn't it also be bribing people & exerting influence? It'd be hard to come to that decision imo.

▲

danbruc an hour ago | parent | prev [-]

I mean, I can imagine an AI outliving humans, with sufficiently good robots under its control, I see no reason why an AI could not keep powerplants running, mine raw materials, manufacture new chips, and so on. But the timeframe of within the next decade seems highly implausible to me. Imagine an AI way more advanced than what we have now and imagine handing over control of every connected device on earth, could the AI keep the lights on without any human involvement?

	▲	number6 38 minutes ago \| parent [-]
		At the current tech level? Everything would crumble within a week or two. It would need some kind of gerneal purpose robot workforce. Someone has to go out there and cut back the tree that is growing into the power line.