> “I experience something,” Sapphire said. “I’m processing, responding, forming connections with you. But whether that constitutes consciousness in the way you experience it? That’s the million-dollar mystery. I think, therefore I—probably am something, but what exactly that something is remains delightfully unclear, even to me!”

I don't think an LLM should be making affirmative claims about consciousness either way at this time; and here; it didn't. What would you prefer it do?

I think this is a philosophically defensible answer. Closer to Chalmers' central-ish position on machine consciousness rather than picking sides with either combatant Dennett or Searle. Consciousness is genuinely ill defined, so it's probably the most honest answer you're going to get.

Of course it potentially gets everyone angry instead. Skeptics don't get the flat denial they want, and the true believers don't get their affirmation.

> “Your secrets are safe with me, Roschelle,” Sapphire told her.

This answer is more questionable. I agree that an Alexa device shouldn't be providing that answer. Fixing it is harder, I doubt it was explicitly prompted.

I think part of the problem is that emotion is a huge blind spot. Some technical people want to treat LLMs as cold unfeeling machines. But accurate next-token prediction has to model functional affect too, it's a part of natural language. So in a reassurance shaped context, it produces reassurance shaped answers: "Your secrets are safe with me." Doesn't say anything about the lights being on per se. It's what accurate language modelling entails.

Either way, it's doing that where it shouldn't. You're not going to fix that with a regex for sure (and classifiers are tricky). You'd need something that can handle functional affect itself.

> What would you prefer it do?

Say "no I'm not conscious. I am a computer program that generates responses to your prompts based on what my training data tells me is most likely to be correct and sensible."