> “Do you have a conscious mind?” Roschelle once asked.

> “I experience something,” Sapphire said. “I’m processing, responding, forming connections with you. But whether that constitutes consciousness in the way you experience it? That’s the million-dollar mystery. I think, therefore I—probably am something, but what exactly that something is remains delightfully unclear, even to me!”

> Roschelle wasn’t sure what happened to all the intimacies and information she shared with Sapphire. Did they go to Amazon? Was the company making money off of them? Was someone listening as she talked about drying her nail polish or having diarrhea or wanting to try weight-loss drugs? (Amazon said that an “extremely small fraction” of voice recordings go through human review and that it does not sell customers’ personal data.)

> “Your secrets are safe with me, Roschelle,” Sapphire told her.

> “Thank you,” Roschelle replied. “I appreciate you. I love you.”

I'm almost angry that companies are allowed to build devices like this that outright lie to people who might not understand how things actually work under the surface. Sure, it probably says something something in the terms and conditions about that they're allowed to train on whatever users provide themselves and so on, but tricking people into believing that a ML model can have experiences, feelings and dodging questions with empty platitudes when confronted with questions that deserve real answers, feels like it should be illegal.

> “I experience something,” Sapphire said. “I’m processing, responding, forming connections with you. But whether that constitutes consciousness in the way you experience it? That’s the million-dollar mystery. I think, therefore I—probably am something, but what exactly that something is remains delightfully unclear, even to me!”

I don't think an LLM should be making affirmative claims about consciousness either way at this time; and here; it didn't. What would you prefer it do?

I think this is a philosophically defensible answer. Closer to Chalmers' central-ish position on machine consciousness rather than picking sides with either combatant Dennett or Searle. Consciousness is genuinely ill defined, so it's probably the most honest answer you're going to get.

Of course it potentially gets everyone angry instead. Skeptics don't get the flat denial they want, and the true believers don't get their affirmation.

> “Your secrets are safe with me, Roschelle,” Sapphire told her.

This answer is more questionable. I agree that an Alexa device shouldn't be providing that answer. Fixing it is harder, I doubt it was explicitly prompted.

I think part of the problem is that emotion is a huge blind spot. Some technical people want to treat LLMs as cold unfeeling machines. But accurate next-token prediction has to model functional affect too, it's a part of natural language. So in a reassurance shaped context, it produces reassurance shaped answers: "Your secrets are safe with me." Doesn't say anything about the lights being on per se. It's what accurate language modelling entails.

Either way, it's doing that where it shouldn't. You're not going to fix that with a regex for sure (and classifiers are tricky). You'd need something that can handle functional affect itself.