> LLMs have seen humans act like conscious beings all over their training data because humans acting like conscious beings IS their training data.

How do we know that humans don't learn how to act conscious by observing other humans who act conscious?

Consciousness doesn't have a precise definition, but if you ask someone to describe it, there is a good chance that the description will include the concept of internal monologue.

The problem is that "internal" monologue is completely meaningless if you never heard an external monologue.

Also, people usually describe internal monologue as something that uses language and language is impossible to learn without communicating with other humans or at least observing other humans.

What I'm saying is that "well, LLM just pretends to be conscious, because it observed humans acting like conscious beings" doesn't really helps us to create a meaningful distinction between human consciousness and machine "consciousness", because same can be argued about us.

We don't know if feral children [0] are conscious and we don't know how to check it.

[0] https://en.wikipedia.org/wiki/Feral_child

    How do we know that humans don't learn how to act conscious by observing other humans who act conscious?