> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with.

It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about how neurons receive information and act on it, but after that we get too tired and give up on how the brain works and treat it like a miracle.

It’s like we desperately want to believe our consciousness is not just electrical impulses in our brain, and we want to ascribe agency and uniqueness to the physical processes going on in our head.

There's definitely a sizable contingent of people who desperately want to believe consciousness is just electrical impulses in our brain. Because "what else could it be"? The fact is that we just don't know, and "abiding in the not-knowing" is for many the most uncomfortable thing ever. Especially for the curious- and rational-minded people this forum tends to attract. I'm one of them, too.

Maybe we do. I think it's a human tendency at large to ascribe pattern or intelligence or spirit where there is only noise. If we can't even prove our own intelligence, doesn't that reinforce the idea that we're in no position to claim intelligence has emerged by running our own intellectual output through a fixed set of weights, the training of which we also designed? At best, any such intelligence would be entirely self-refential and exposed to the question of whether we ourselves are intelligent. If your position is that we are not, then there's no way an LLM could be.

> but after that we get too tired and give up on how the brain works and treat it like a miracle.

I disagree. We know very well how neurons work, and we have a pretty good idea of how neural activity translates to behavior. In other words, we have a pretty good idea on how the brain works. We stop at consciousness because as of yet it is in the realm of philosophy, not science. We don‘t know what consciousness is or even whether or not it is useful for science and we are simply waiting for the philosophers guides us out of that situation.

Note that both cognitive psychology and behavioral psychology has done fine without tackling consciousness. When neuropsychology emerged in the 1980s it complemented both these fields perfectly. The situation is the opposite with the philosophy of mind which grew significantly around the same time.

There have been some attempts to describe consciousness as an emerging phenomena out of neural activity, but so far all of these attempts have failed, or at least failed to turn consciousness into a useful term in psychology (the way gravity is a useful term in physics). I think it is equally likely that these attempts have failed because consciousness may simply not be a useful term in psychology, that is as likely as it is that we simply don‘t understand it well enough.