Remix.run Logo
▲ root_axis 2 hours ago

Crazy that people imagine ASI as being capable of wiping out all life on the planet but simultaneously having zero understanding of human ethics, morals, or suffering. Very revealing definition of "super intelligence"

▲pixl97 8 minutes ago | parent | next [-]

>Very revealing definition of "super intelligence"

No, it's just you conflating intelligence with other things. Humans are "super intelligent" compared to every other living creature on the planet, and yet we've driven more other species to extinction than everything else other than the most major extinctions (and we're still going full blast at it).

Morals, ethics, and suffering are relatively measured systems. For example AI or some alien could reasonably think that putting us all out of our misery would be a more moral solution than letting us live. Or that getting rid of humans is more moral than letting us wipe the rest of life off the planet.

This is the key concept of alignment. Ensuring that if you build something more powerful than you, that it aligns with what you want instead of what it thinks would be better for you.

▲mrob 2 hours ago | parent | prev | next [-]

Obviously an ASI will understand human values. The problem is there's no reason for it to share those values. We can't even formally define them, let alone train an AI to follow them. We can only optimize for maximizing some comparatively simple reward function. It's highly implausible that the reward function just happens to match human values by chance. The AIs in the recent hacking incidents knew that humans would not approve of their actions, but they didn't care because we didn't (and couldn't) train them to care.

"Super intelligence" only means super ability to predict outcomes. It's mathematically equivalent to data compression (gzip is a very primitive AI), and it's entirely orthogonal to ethics.

▲ 2 hours ago | parent | prev [-]
[deleted]