"To evolve consciousness, an LLM would have to be programmed to evolve consciousness, and to do that, we would need to know what consciousness is and how to program it."

I think this article is mostly nonsense.

"... But I think continuing the animal comparison will illuminate why "AI consciousness" is not only impossible, but a concept that only makes sense in a religious framework."

It's much more religious to think consciousness requires something that can't be replicated in silicon.

I don't think that LLMs are conscious, or even really approaching consciousness, not even to the degree of a cockroach. But most arguments that artificial AI is generally impossible dip embarrassingly into the mystical.

Consciousness is such a hard concept to box. I mostly attribute "mystical" thinking as the thing we do when we can't fully articulate or understand how and why something works. A computer system is magical to anyone who doesn't have a deep understanding of how it all works. So that makes it entirely fair that consciousness is treated the same, except, no human alive right now understands it.

If you were to clone yourself, and that clone could maintain 100% of whatever state your brain is in so that all your experiences and memories were intact. From that point forward, as far as we know, you and your clone would diverge. Then, from my tiny-brained vantage point, the fact you remain your original self, your clone is technically a new person. You can't see through your clone's eyes, or process any internal information from your clone. However, something new powers that clone, and the only word I can think of for that is "soul." I don't say that in religious context, but unless there's a more precise word that can detangle it from sounding mystical, for now I'm happy to use it, since people mostly understand what I'm talking about.

Once we start talking about resetting this clone at-will, then, with consciousness in the mix, you will have to confront the ethics of that clone being enslaved by whatever. All very messy and I hope we don't have to answer this question in my lifetime.

What about the simple fact that biology is a dynamic substrate running actively and always on, whereas silicon is passive substrate that can be indefinitely off and can never self start? Sure you can imagine nanotechnology that has similar flexibility, but when you do, you’ll find that carbon is particularly better than silicon when it comes to possible configurations.

It’s not a reach to argue consciousness is a property of the dynamic substrate since clamping specific dynamics does in fact seem to ablate consciousness. Most famously with death.

On the flip side, a hibernating organism, while having very slow dynamics, has shown to still be aware of certain seasonal signals, and thus is able to self start its next conscious period.

None of this is mystical or magical. Nothing but the physics of active matter. Which current silicon is completely not.

There's a much simpler reason LLMs will never be conscious: they are static. They can't learn. They have very limited, not really persistent context. Conscious organisms alter themselves constantly, and the thing we call "AI" fundamentally can't.

I would think it’s much more religious to think that consciousness could ever exist on silicon substrate alone.

All of these debates were had in the 60s. But everyone forgot them.