I get the sense that he is misidentifying the potential locus of consciousness..

In the same way that the sound waves and facial expressions I produce are not conscious, the output json of an LLM is obviously not conscious either.

The locus of consciousness and subjective experience may be in the computer, either at inference time or training time..

If we ask "what is it conscious of when it writes something" then training time is irrelevant.

The software that does the inference is clearly just computer code.

What we're left with is a fictional character being briefly conscious while its dialog is being written, which is pretty absurd.