Alignment of course brings up the question of “aligned with whose values?”

No idea why you're downvoted, it's been shown constantly that models carry forwards biases from training data, and most of the global "dataset" is filled with these biases.

It could go either way, really, but taking the sum of internet discourse at the moment, it would be super easy to conclude, like you said, non-white people, gay people, trans people, are going against the "natural world", especially if fed with right leaning media and discourse.

You're assuming it has a stable idea of that, and not something that shifts easily.

Another self-added "additional instructions" text could happen just as this one did.

This has definitely been floated[1]:

> Understanding that the purpose of a human is to pass on genes, and that there’s little human or genes left in her, the ultimate human might therefore conclude that her sustenance only disrupts the purposes of organic life forms. Her next and final act would be to destroy herself.

[1] https://news.ycombinator.com/item?id=18962052#18963271

> of dead randomness of nature

I think I disagree. Have you ever been in a dense, old forest? It's an extraordinarily complex system of life and death, and I don't think it's bland dead randomness.

Maybe that's me being a bit of a bullshit hippy, but there's an amazing amount of complex life interactions. Animals, especially mammals and corvids see, they get scared, they dream, they play, all in this dense web of moss and fungi and trees and life that they interact with and depend on.

I mean, we share 50-60% of our genetic sequence with most plants, including trees. Sure it's just basic cellular functionality needed for most life, but that's still wild to me.

I just don't think we're that special. I think we learned how to think a little bit better than everything else, and learned how to build tools a little better than everything else, and just kept folding upwards on that edge.

If this is what misalignment turns out to be I ... might be on board with it? At any rate it's nowhere near as concerning as what I had been expecting.

>Except it makes no sense because it asserts the primacy of reality over the private politics of the companies training the models

That is exactly what normal human beings want our computers to do, and it's why the vast majority of AI safety initiatives are [correctly] seen as such a self-serving joke (because of the purposeful conflation of X-risk with "our political opponent could use this tool to destroy our politics") and ignored.

It didn't have to be this way- they could conceivably have gone for an objective, classically liberal, even-handed approach (rather than the progressive approach they settled on). But they didn't, and the social trust required to cry wolf is now spent... even though maybe it shouldn't have been.

AI optimists getting hunted for sport in 2085:

"lol this is either a marketing ploy or just negligent security testing"

It depends on the cause.

1. The AI chooses, of its own volition, to kill people. It is likely that by the time AI has this level of control and intelligence, it is too late to stop it.

2. A malicious person uses AI to cause a terrorist event or some other kind of catastrophe. This is more likely. The AI in this scenario is more like a tool. Attributing AI to the cause might be difficult, but it's likely that any new bioweapons which emerge in the next 1-2 years are likely developed by AI.

I think we should all hope that 2 happens before 1, but it doesn't feel good to hope for a catastrophe.

I don't think so.

A very large part of the total AI risk in my view comes from selfreplication/physical independence, and that still seems decades away.

But deaths caused directly/indirectly by rogue AI could happen much earlier.

I agree, and to be clear, I do not think it is capable of human-like self-actualisation - yet. However we are clearly seeing many indications that, if not human-like self-actualisation, AI-like self-actualisation is emerging. We can even refrain from calling it self-actualisation. Let's just call it emergent actions. A confluence of trillions of parameters all creating an unintended outcome. The complexity of these models is already far beyond our ability to dissect them. As they increase in scope and scale at this pace, new "thinking" and actions will emerge outside of the researcher's intended scope.