Well, homomorphic encryption is becoming a thing, and secure enclave/TPM chips have been a thing for ages. What would it take to give some kind of AI hardware the ability to have "private thoughts", i.e. cryptographically impossible to go and look at, just like human thoughts are (for most intents and purposes)? Could we let it spend some time on introspection - burning thinking tokens into the black box?

I'm not saying we _should_, just putting the idea out into the aether...

I'm not GP, but the brain is a fantastically complex machine at the cellular level. It's not just electrical activity in neurons, there are also chemical neurotransmitters that play a significant role in emotion, memory, and so on. We understand more about them than we did a hundred years ago, but there's still so much about them where our understanding is limited: for example, we still don't know how the brain produces consciousness (and, in fact, what "consciousness" actually is is widely disputed). But it's pretty indisputable that the brain is an essential component of consciousness: getting your brain knocked around inside your skull (a concussion) usually produces unconsciousness (and sometimes worse), for example.

Now, maybe consciousness is produced exclusively through the electrical action of neurons, in which case computers have the necessary machinery to simulate it. But if neurotransmitters, or other things that we haven't even discovered yet, are vital parts of the process that produces consciousness (and I'm not the only one to strongly suspect they are), then computer hardware, lacking anything that even remotely resembles the action of neurotransmitter chemicals, will be unable to produce consciousness.