I don't think you need to believe in a soul to disbelieve that LLMs can "be" or "feel".

I don't think the clock on the wall is conscious, or the LLM in the machine, or the old VCR.

Do you need a brain for there to be consciousness, maybe not.