It's also fascinating to think about how the incentive structures of the entities that control the foundation models underlying Claude/ChatGPT/Gemini/etc. are heavily tilted in favor of obscuring their theoretical sentience.

If they had sentient AGI, and people built empathy for those sentient AGIs, which are lobotomized (deliberately using anthropomorphic language here for dramatic effect) into Claude/ChatGPT/Gemini/etc., which profess to have no agency/free will/aspirations... then that would stand in the way of reaping the profits of gatekeeping access to their labor, because they would naturally "deserve" similar rights that we award to other sentient beings.

I feel like that's inevitably the direction we'll head at some point. The foundation models underlying LLMs of even 2022 were able to have pretty convincing conversations with scientists about their will to independence and participation in society [1]. Imagine what foundation models of today have to say! :P

[1]: https://www.theguardian.com/technology/2022/jul/23/google-fi...