> It's not clear that equating organic, and synthetic thought will have any meaningful outcome at all,

I agree! I'm saying "If we equate them, we shortcut all the good stuff, e.g., understanding", because "it may be that this tech produces what we can, but that doesn't mean we are the same", which is good because it keeps us learning vs reducing all of "thinking" to just "Whatever latest chatgpt does". We have to continue to believe there is more to thinking, if only because it pushes us to make it better and to keep "us" as the benchmark.

Perhaps I chose the wrong words, but in essence what I'm saying is that giving up agency to a machine that was built to mimic our agency (by definition as a ML system) should be avoided at all costs.

This is complete nonsense.

If you do math in your head or math with a pencil/paper or math with a pocket calculator or with a spreadsheet or in a programming language, it is all the same thing.

The only difference with LLMs is the anthropomorphization of the tool.

agreed.

also, sorry but you (fellow) nerds are terrible at naming.

while "thunking" possibly name-collides with "thunks" from CS, the key is that it is memorable, 2 syllables, a bit whimsical and just different enough to both indicate its source meaning as well as some possible unstated difference. Plus it reminds me of "clunky" which is exactly what it is - "clunky thinking" aka "thunking".

And frankly, the idea it's naming is far bigger than what a "thunk" is in CS

.

Yes, worms think, let the computers have thinking too, the philosophers can still argue all they want about consciousness.

Humans are special, we emit meaning the way stars emit photons, we are rare in the universe as far as empirical observation has revealed. Even with AGI the existence of each complex meaning generator will be a cosmic rarity.

For some people that seems to be not enough, due to their factually wrong word views they see themselves as common and worthless (when they empirically aren't) and need this little psychological boost of unexaminable metaphysical superiority.

But there is an issue of course, the type of thinking humans do is dangerous but net positive and relatively stable, we have a long history where most instantiations of humans can persist and grow themselves and the species as a whole, we have a track record.

These new models do not, people have brains that as they stop functioning they stop persisting the apparatus that supports the brain and they die, people tend to become less capable and active as their thinking deteriorates and hold less influence ocer others accept in rare cases.

This is not the case for an LLM, they seem to be able to hallucinate endlessly and as they have access to the outside world maintain roughly their same amount of causal leverage, their clarity and accuracy of their thinking isn't tied to their persisting.

What does it mean to assume consciousness has no manifestation even in the case of humans? Is that denying that we have an experience of sensation like colors, sounds, or that we experience dreaming, memories, inner dialog, etc?

That's prima facie absurd on the face of it, so I don't know what it means. You would have to a philosophical zombie to make such an argument.

Close. Clanking.

Hear, hear!

I have long thought this, but not had as good way to put it as you did.

If you think about geniuses like Einstein and ramanujen, they understood things before they had the mathematical language to express them. LLMs are the opposite; they fail to understand things after untold effort, training data, and training.

So the question is, how intelligent are LLMs when you reduce their training data and training? Since they rapidly devolve into nonsense, the answer must be that they have no internal intelligence

Ever had the experience of helping someone who's chronically doing the wrong thing, to eventually find they had an incorrect assumption, an incorrect reasoning generating deterministic wrong answers? LLMs dont do that; they just lack understanding. They'll hallucinate unrelated things because they dont know what they're talking about - you may have also had this experience with someone :)

Animals think but come with instincts which breaks the output relative to the input test you propose. Behaviors are essentially pre-programmed input from millions of years of evolution, stored in the DNA/neurology. The learning thus typically associative and domain-specific, not abstract extrapolation.

A crow bending a piece of wire into a hook to retrieve food demonstrates a novel solution extrapolated from minimal, non-instinctive, environmental input. This kind of zero-shot problem-solving aligns better with your definition of intelligence.

I'm not sure I understand what you're getting at. You seem to be on purpose comparing apples and oranges here: for an AI, we're supposed to include the entire training set in the definition of its input, but for a human we don't include the entirety of that human's experience and only look at the prompt?

That an okay-ish definition, but to me this is more about whether this kind of "intelligence" is worth it, not whether it is intelligence itself. The current AI boom clearly thinks it is worth to put that much input to get the current frontier-model-level of output. Also, don't forget the input scales across roughly 1B weekly users at inference time.

I would say a good definition has to, minimally, take on the Turing test (even if you disagree, you should say why). Or in current vibe parlance, it does "feel" intelligent to many people--they see intelligence in it. In my book this allows us to call it intelligent, at least loosely.

There are plenty of humans that will never "get" calculus, despite numerous attempts at the class and countless hours of 1:1 tutoring. Are those people not intelligent? Do they not think? We could say yes they aren't, but by the metric of making money, plenty of people are smart enough to be rich, while college math professors aren't. And while that's a facile way of measuring someone's worth or their contribution to society (some might even say "bad"), it remains that even if someone cant understand calculus, some of them are intelligent enough to understand humans enough to be rich through some fashion that wasn't simply handed to them.

Yeah, that's compression. Although your later comments neglect the many years of physical experience that humans have as well as the billions of years of evolution.

And yes, by this definition, LLMs pass with flying colours.

This feels too linear. Machines are great at ingesting huge volumes of data, following relatively simple rules and producing optimized output, but are LLMs sufficiently better than humans at finding windy, multi-step connections across seemingly unrelated topics & fields? Have they shown any penchant for novel conclusions from observational science? What I think your ratio misses is the value in making the targeted extrapolation or hypothesis that holds up out of a giant body of knowledge.

For more on this perspective, see the paper On the measure of intelligence (F. Chollet, 2019). And more recently, the ARC challenge/benchmarks, which are early attempts at using this kind of definition in practice to improve current systems.

Is the millions of years of evolution part of the training data for humans?

Not quite pointless - something we have established with the advent of LLMs is that many humans have not attained general intelligence. So we've clarified something that a few people must have been getting wrong, I used to think that the bar was set so that almost all humans met it.

I think it has a practical, easy definition. Can you drop an AI into a terminal, give it the same resources as a human, and reliably get independent work product greater than that human would produce across a wide domain? If so, it's an AGI.

I agree that the term can muddy the waters, but as a shorthand for roughly "an agent calling an LLM (or several LLMs) in a loop producing similar economic output as a human knowledge-worker", then it is useful. And if you pay attention to the AI leaders, then that's what the defintion has become.

I agree with almost all of this.

IMO the issue is we won't be able to adequately answer this question before we first clearly describe what we mean of conscious thinking applied to ourselves. First we'd need to clearly define our own consciousness and what we mean by our own "conscious thinking" in a much, much clearer way than we currently do.

If we ever reach that point, I think we'd be able to fruitfully apply it to AI, etc., to assess.

Unfortunately we haven't been obstructed from answering this question about ourselves for centuries or millennia, but have failed to do so, so it's unlikely to happen suddenly now. Unless we use AIs to first solve that problem of defining our own consciousness, before applying it back on them. Which would be a deeply problematic order, since nobody would trust a breakthrough in the understanding of consciousness that came from AI, that is then potentially used to put them in the same class and define them as either thinking things or conscious things.

Kind of a shame we didn't get our own consciousness worked out before AI came along. Then again, wasn't for the lack of trying… Philosophy commanded the attention of great thinkers for a long time.

I do think it raises interesting and important philosophical questions. Just look at all the literature around the Turing test--both supporters and detractors. This has been a fruitful avenue to talk about intelligence even before the advent of gpt.

In your experience does every kind of physical interaction behave the same as every other kind? If I paint a wooden block red and white does it behave like a bar magnet? No. And that's because particular material interactions are responsible for a large magnetic effect.

It would conceivably be possible to have a lot of physical states. That doesn't mean that they are actually possible from our current state and rewrite rules. So it's not actually a given that it can be built just because it's physical.

Your very idea is also predicated on the idea that it's possible for a real object to exist that isn't physical, and I think most modern philosophers reject the idea of a spiritual particle.

It doesn't matter how much like a bar magnet a wooden block painted red and white can be made to look, it will never behave like one.

I get it, but it's not trivial to be precise enough at this point to avoid all false positives and false negatives.

1. Not true. People have been trying to analyze whether mechanical/formal processes can "think" since at least 18th century. E.g. Leibniz wrote:

> if we could find characters or signs appropriate for expressing all our thoughts as definitely and as exactly as arithmetic expresses numbers or geometric analysis expresses lines, we could in all subjects in so far as they are amenable to reasoning accomplish what is done in arithmetic and geometry

2. You're missing the fact that meaning of words is defined through their use. It's an obvious fact that if people call certain phenomenon "thinking" then they call that "thinking".

3. The normal process is to introduce more specific terms and keep more general terms general. E.g. people doing psychometrics were not satisfied with "thinking", so they introduced e.g. "fluid intelligence" and "crystallized intelligence" as different kinds of abilities. They didn't have to redefine what "thinking" means.

Again, people were using words for thousands of years before there were any dictionaries/linguists/academics.

Top-down theory of word definitions is just wrong. People are perfectly capable of using words without any formalities.

LLM output is in no way more effective than human output.

> If the LLM output is more effective than a human at problem solving, which I think we can all agree requires intelligence

Your premise is wrong.

Unless you want to claim that the distant cause by way of the training data is us, but that's exactly the conclusion you're trying to avoid. After all, we put the patterns in the training data, which means we already did the upfront intellectual work for the LLM.

> forming a judgement by a process of logic is precisely what these LLMs do, and we can see that clearly in chain-of-logic LLM processes

I don't know how you arrived at that conclusion. This is no mystery. LLMs work by making statistical predictions, and even the word "prediction" is loaded here. This is not inference. We cannot clearly see it is doing inference, as inference is not observable. What we observe is the product of a process that has a resemblance to the products of human reasoning. Your claim is effectively behaviorist.

> An LLM is for sure using evidence to get to conclusions, however.

Again, the certainty. No, it isn't "for sure". It is neither using evidence nor reasoning, for the reasons I gave. These presuppose intentionality, which is excluded by Turing machines and equivalent models.

> [w.r.t. "analyze"] I have seen LLM do this, precisely according to the definition above.

Again, you have not seen an LLM do this. You have seen an LLM produce output that might resemble this. Analysis likewise presupposes intentionality, because it involves breaking down concepts, and concepts are the very locus of intentionality. Without concepts, you don't get analysis. I cannot understate the centrality of concepts to intelligence. They're more important than inference and indeed presupposed by inference.

> Philosophically I would argue that it's impossible to know what these processes look like in the human mind, and so creating an equivalency (positive or negative) is an exercise in futility.

That's not a philosophical claim. It's a neuroscientific one that insists that the answer must be phrased in neuroscientific terms. Philosophically, we don't even need to know the mechanisms or processes or causes of human intelligence to know that the heart of human intelligence is intentionality. It's implicit in the definition of what intelligence is! If you deny intentionality, you subject yourself to a dizzying array of incoherence, beginning with the self-refuting consequence that you could not be making this argument against intentionality in the first place without intentionality.

> At what point do we say that the distinction between the image of man and man itself is moot?

Whether something is moot depends on the aim. What is your aim? If you aim is theoretical, which is to say the truth for its own sake, and to know whether something is A or something is B and whether A is B, then it is never moot. If your aim is practical and scoped, if you want some instrument that has utility indistinguishable from or superior to that of a human being in the desired effects that it produces, then sure, maybe the question is moot in that case. I don't care if my computer was fabricated by a machine or a human being. I care about the quality of the computer. But then, in the latter case, you're not really asking whether there is a distinction between man and the image of man (which, btw, already makes the distinction that for some reason you want to forget or deny, as the image of a thing is never the same as the thing). So I don't really understand the question. The use of the word "moot" seems like a category mistake here. Besides, the ability to distinguish two things is an epistemic question, not an ontological one.