"To evolve consciousness, an LLM would have to be programmed to evolve consciousness, and to do that, we would need to know what consciousness is and how to program it."

I think this article is mostly nonsense.

"... But I think continuing the animal comparison will illuminate why "AI consciousness" is not only impossible, but a concept that only makes sense in a religious framework."

It's much more religious to think consciousness requires something that can't be replicated in silicon.

I don't think that LLMs are conscious, or even really approaching consciousness, not even to the degree of a cockroach. But most arguments that artificial AI is generally impossible dip embarrassingly into the mystical.

It would be helpful to define "consciousness". I'm not convinced that i as a human have this "consciousness" thing. In fact, I strongly suspect that i am merely a statistical machine picking words from a learned distribution.

Since it is currently impossible for us to transfer our consciousness into any other entity, we may never be able to determine whether another entity truly possesses consciousness. If a living being no longer responds to the outside world in any way, can we conclude that it is unconscious—even if its “soul” still exists, screaming in agony within its own mind? Yes, much like the way we look at someone in a vegetative state. Zhuangzi said, “You are not a fish—how can you know what makes a fish happy?” Humanity longs for the emergence of another intelligent being, fearing that we are alone in the endless universe and destined to march toward extinction. Yet perhaps we are also secretly pleased that no other intelligent beings have appeared, because this fits perfectly with the belief that human beings are God’s perfect creation—a belief that gives humanity its arrogance. Could our rejection of the possibility that AI possesses consciousness likewise be a kind of arrogant, subconscious exclusionism?

On the about page, I see a strong political/religious stance, but no qualifications.

"My main objection to generative AI is from morality—it harms people, and there isn't even a trade-off to debate, because every purported benefit of genAI is corporate hype and/or science fiction."

Perhaps the belief that AI is inherently non-conscious will one day become a kind of religion—an anthropocentric one. That’s not to say I believe today’s AI systems are conscious in anything like the human sense.

I don't disagree with the title of the article but I also don't think you can prove anything about consciousness with language. How consciousness can arise from matter is still a very much a mystery.

I think the whole article is undermined by this paragraph:

> We don't know what consciousness is. We don't know why we have it. We don't know why some animals have it and others don't. We don't know if it can be quantified. We don't have a model of consciousness or know if one can be created.

I agree that we don't know what consciousness is and can't quantify it, and yet the rest of the article is… a quantification of consciousness as a human evolutionary trait that cannot emerge through genetic algorithms or LLM training.

Genetic algorithms certainly differ from human evolution in numerous ways, but I don't agree with the fundamental distinction the article makes—that evolutionary algorithms are constrained while human evolution is not. The 2D cars can't "hack the simulation and turn the gravity down" any more than humans can hack the laws of physics and turn the gravity on Earth down. Yet humans attained consciousness.

I do not know whether LLMs are conscious or not, but I do firmly believe it's possible to build a machine that is conscious: after all, humans are conscious and, unless one subscribes to the kind of religious thinking the article derides, a perfect simulation of the physical workings of a human would thus also be conscious. To my knowledge, nothing we've seen so far in physics rules out such a simulation one day existing.

I also suspect that, once we do manage to make a machine that is conscious, we'll spend inordinate time and effort trying to convince ourselves it's not in order to justify the ways we treat it.

The first season of the tv show Westworld is a treasure trove of thought experiments exploring the value of what extending conciousness to "machines" would look like and I'm pretty sure there's absolutely zero value for actually building it.

To me this is like deja-vu: Ethology started out with "don't anthropomorphize" which is correct, but people read that to mean that animals can't think. Each paper down the line showed that they CAN think, but that each species and even individual have their own ways of thinking.

We know that gorillas think and communicate. Many papers after Koko can tell you that much. Her skill (or lack thereof) at human communication is not quite the revelation that people think it is. Because she was a Gorilla.

Meanwhile some of the messaging from big AI companies is truly atrocious; let's admit it. People obviously want to push back, and that needn't be wrong. Here though, the instinct to push back threatens to veer into cargo-cult skepticism. I think the question of whether or not LLMs can or cannot have human consciousness is a category error. Because they're LLMs.

How is it that sign language association learning is different from spoken language learning? What about humans using simple sign language in their early life?

Is it not possible that we try to detract too much here?

But I don't want Claude to become conscious. I want Claude to help me write good code.

> To evolve consciousness, an LLM would have to be programmed to evolve consciousness, and to do that, we would need to know what consciousness is and how to program it.

Since consciousness only counts when it is formed deliberately rather than accidentally. Though that would mean…

[dead]

[dead]