On one hand, we don't have any idea what consciousness is or how it happens. For all we know, putting a ton of numbers onto a graphics card and doing matrix math on them is enough to make it.

On the other hand, this really feels like getting freaked out about seeing a realistic photo of a person for the first time, because it looks so much like a person, or hearing a recording of someone speaking for the first time because it sounds like they're really there. They're reproductions of a person, but they are not the person. Likewise, LLMs seem to me to be reproductions of thought, but they are not actually thought.