I wouldn't put too much emphasis on the "I can't prove they aren't" part. Having taken a few philosophy classes in college, I quickly realized that you can't actually prove that solipsism is wrong. I can't actually prove that everything I see, every person I talk to, isn't just a figment of my imagination. And yet, I categorically reject solipsism anyway, as any sane person should. The world is real, other people are real conscious beings, and so on. To believe otherwise is to literally be insane. And yet, any argument you could come up with for rejecting solipsism, a solipsist would reject: "Oh, that's just a figment of my imagination going off on a tangent again."

Just because you can't prove an assertion doesn't mean it's wrong.

We have to define AGI first. Then depending on the definition we may be able recognize it when we make it. Most would probably think a definition of AGI that we couldn’t recognize would not be a great definition.

Also AGI almost certainly does not equal able to suffer. For example animals can suffer but would probably not meet the definition of AGI. we could create things that feel suffering even if they do not meet the definition of AGI.

So a better question might be: “can we detect if we create a system that can suffer?” And just drop the AGI part entirely. Then that gets us back to defining “suffering”. Agreeing on the definition always seems to be the hard part…

I think it’s also important to disaggregate consciousness from suffering. We could create a conscious system with no ability to suffer (or no ability to suffer in a way we would recognize as suffering). Most of what causes us suffering (loneliness, pain, failure) is driven by emotional feedback granted to us by evolution. Without the same evolutionary pressures or a similar cognitive structure, even a conscious AGI might not experience suffering

On that note, would we recognise the second coming of Jesus?

How many followers on instaX would they need to have for folks too truly believe it’s theRealJesus returning to save us.

It might sound silly, but at this stage, the belief in a benevolent AGI or Jesus are very similar. In both cases its us delegating our responsibilities to a third party and us hoping that third party will fix our problems for us.

Thankfully Hollywood invented happy ends for just such a scenario.

There is nothing wrong with murdering a million lives while iteratively developing – it's how we grow and study bacteria after all. Any person that fixed their gut by sticking to a diet also did exactly that.

So if we are concerned with the AGI suffering, then we don't need to determine consciousness at all. We just need to prove that they don't have suffering capabilities. Which is a way easier question.

I'd recognize the emotional robot if it was like Marvin the paranoid android.