>Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage.

Obviously, it is different or else we would just use JIRA and a database to replace GPT. Models very obviously do NOT store training data in the weights in the way you are imagining.

>So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking".

Thinking is by all appearances substrate independent. The moment we created computers, we created another substrate that could, in the future think.

You're getting to the heart of the problem here. At what point in evolutionary history does "thinking" exist in biological machines? Is a jumping spider "thinking"? What about consciousness?

This is an overly reductive view of a fully trained LLM. You have identified the pieces, but you miss the whole. The inference code is like a circuit builder, it represents the high level matmuls and the potential paths for dataflow. The data blob as the fully converged model configures this circuit builder in the sense of specifying the exact pathways information flows through the system. But this isn't some inert formalism, this is an active, potent causal structure realized by the base computational substrate that is influencing and being influenced by the world. If anything is conscious here, it would be this structure. If the computational theory of mind is true, then there are some specific information dynamics that realize consciousness. Whether or not LLM training finds these structures is an open question.

A similar point was made by Jaron Lanier in his paper, "You can't argue with a Zombie".

> So if the disk isn't alive, the file on it isn't alive, the inference software is not alive - then what are you saying is alive and thinking?

“So if the severed head isn’t alive, the disembodied heart isn’t alive, the jar of blood we drained out isn’t alive - then what are you saying is alive and thinking?”

- Some silicon alien life forms somewhere debating whether the human life form they just disassembled could ever be alive and thinking

That's not a computation, it's a side effect. It just depends on what you wire your "computer" up to. A Turing machine in itself is just a (potentially non-returning) mathematical function, but you are free to map any input/output to it.

Actually, the way LLMs are extended with tools is a pretty much the same (an LLM itself has no access to the internet, but if it returns some specific symbols, the external "glue" will do a search and then the LLM is free to use the results)

Who truly knows if you can make an if-else + randomness structure big enough to become smart?

But yes, we built a machine that generates text similar to what we built it from, and now we're looking at it generating text and are all impressed.

This is exactly why I mentioned the weather.

A scientific paper has to be verifiable, you should be able to recreate the experiment and come to the same conclusion. It's very very difficult to do with brains with trillions of parameters and that can't be controlled to the neuron level. Nothwithstanding the ethical issues.

We don't have a world weather simulator that is 100% accurate either given the complex interplay and inability to control the variables i.e. it's not verifiable. It'd be a bit silly to say we don't know why it's going to rain at my house tomorrow.

Until then it is a hypothesis, and we can't say we know even if the overwhelming evidence indicates that in fact that we do know.

> "quite robust against disturbances."

does not mean that the essential thing gives rise to consciousness is only approximate. To give an example from software, you can write software is robust against bad input, attempts to crash it, even bit flips. But, if I came in and just changed a single character in the source code, that may cause it to fail compilation, fail to run, or become quite buggy.

> If you don't want that to be true, you need some kind of magic,

This is just what I'm saying is a false dichotomy. The only reason some are unable to see beyond it is that we think the basic logic we understand are all there could be.

In this respect physics has been very helpful, because without peering into reality, we would have kept deluding ourselves that pure reason was enough to understand the world.

It's like trying to explain quantum mechanics to a well educated person or scientist from the 16th century without the benefit of experimental evidence. No way they'd believe you. In fact, they'd accuse you of violating basic logic.

dude u need to do some psychedelics.

I'm not talking about "identical" consciousnesses. I mean the same consciousness. The same consciousness cannot split into two, can it?

Either it is (and continues to be) the same consciousness, or it is not. If it were the same consciousness, then you would have a person who exists in two places at once.

>Is is the cluster that is conscious? Or the individual machines, or the chips, or the algorithm, or something else?

The bound informational dynamic that supervenes on the activity of the individual units in the cluster. What people typically miss is that the algorithm when engaged in a computing substrate is not just inert symbols, but an active, potent causal/dynamical structure. Information flows as modulated signals to and from each component and these signals are integrated such that the characteristic property of the aggregate signal is maintained. This binding of signals by the active interplay of component signals from the distributed components realizes the singular identity. If there is consciousness here, it is in this construct.

I explained the experiments that you would do to figure that out: you modify parts of the system, and check if and how much that affects the consciousness. Paint the interconnects a different color: probably won't affect it. Replace the interconnect protocol with a different one: probably will have some effect. So, the paint on the interconnect: not a part of the consciousness. The interconnect protocol: part of the consciousness. If we are convinced that this is a real consciousness and thus these experiments are immoral, we simply wait until accidents naturally occur and draw conclusions from that, just like we do with human consciousness.

Of course, "the consciousness" is a nebulous concept. It would be like asking "which part of my processor is Windows" to some extent. But it's still fair to say that Windows is contained within my computer, and that the metal framing of the computer is not part of Windows.

It's the most reasonable thing to do because we have no actual way of measuring and knowing. It is still speculation.

We can measure the physiological activity, but not whether it gives rise to the same sensations that we experience ourselves. We can reasonably project and guess that they are the same, but we can not know.

In practical terms it does not matter - it is reasonable for us to act as if others do experience the same we do. But if we are to talk about the nature of conscience and sentience it does matter that the only basis we have for knowing about other sentient beings is their self-reported experience.

You can measure model activity even better.

How do you know that model processing text or image input doesn't go through feeling of confusion or excitement or corrupted image doesn't "smell" right for it?

Just the fact that you can pause and restart it doesn't mean it doesn't emerge.

Ignoring that "cognate" isn't a verb, we have basis for making any claim about the necessity of that coupling.

I'm with you on this.

But asked for reasons I can only point to the social nature of their societies, where love and anger make sense, or of their hurt-behavior.

I also find it very hard to believe that everything else is slow evolution of components, and here all of a sudden something super complex comes into being out of nowhere.

But I still have no idea how it could work. What are the components and their interplay?

I should have been more exact and said sentience vs sapience in animals vs humans.