I think Chiang is right about this, but there is a related philosophical mystery. The trend from Deep Blue to AlphaGo to LLMs solving Erdos problems suggests that Peter Watts was onto something when he wrote Blindsight. Reasoning ability is apparently independent of consciousness?

We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?

▲

david-gpu 2 hours ago | parent [-]

We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.

	▲	search_facility 2 hours ago \| parent [-]
		Well, at the very least we have a base to distinct different kinds then. Probably the main problem of people implying LLM consciousness is that they imply LLM have human-kind of consciousness. Judging only on "how they speak", generally, they insist on using the same word that labels human consciousness (exclusively), etc. But there are so many instrinic differences that such claim is not feasible despite similar "talking abilities".