| ▲ | ivraatiems 3 days ago |
| The author searches for a midpoint between "AIs are useless and do not actually think" and "AIs think like humans," but to me it seems almost trivially true that both are possible. What I mean by that is that I think there is a good chance that LLMs are similar to a subsystem of human thinking. They are great at pattern recognition and prediction, which is a huge part of cognition. What they are not is conscious, or possessed of subjective experience in any measurable way. LLMs are like the part of your brain that sees something and maps it into a concept for you. I recently watched a video on the creation of AlexNet [0], one of the first wildly successful image-processing models. One of the impressive things about it is how it moves up the hierarchy from very basic patterns in images to more abstract ones (e. g. these two images' pixels might not be at all the same, but they both eventually map to a pattern for 'elephant'). It's perfectly reasonable to imagine that our brains do something similar. You see a cat, in some context, and your brain maps it to the concept of 'cat', so you know, 'that's a cat'. What's missing is a) self-motivated, goal-directed action based on that knowledge, and b) a broader context for the world where these concepts not only map to each other, but feed into a sense of self and world and its distinctions whereby one can say: "I am here, and looking at a cat." It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical. I think LLMs represent a huge leap in technology which is simultaneously cooler than anyone would have imagined a decade ago, and less impressive than pretty much everyone wants you to believe when it comes to how much money we should pour into the companies that make them. [0] https://www.youtube.com/watch?v=UZDiGooFs54 |
|
| ▲ | vidarh 3 days ago | parent | next [-] |
| > or possessed of subjective experience in any measurable way We don't know how to measure subjective experience in other people, even, other than via self-reporting, so this is a meaningless statement. Of course we don't know whether they are, and of course we can't measure it. I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it. > What they are not is conscious And this is equally meaningless without your definition of "conscious". > It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical. Unless we can find indications that humans can exceed the Turing computable - something we as of yet have no indication is even theoretically possible - there is no rational reason to think it can't. |
| |
| ▲ | ivraatiems 3 days ago | parent | next [-] | | > Unless we can find indications that humans can exceed the Turing computable - something we as of yet have no indication is even theoretically possible - there is no rational reason to think it can't. But doesn't this rely on the same thing you suggest we don't have, which is a working and definable definition of consciousness? I think a lot of the 'well, we can't define consciousness so we don't know what it is so it's worthless to think about' argument - not only from you but from others - is hiding the ball. The heuristic, human consideration of whether something is conscious is an okay approximation so long as we avoid the trap of 'well, it has natural language, so it must be conscious.' There's a huge challenge in the way LLMs can seem like they are speaking out of intellect and not just pattern predicting, but there's very little meaningful argument that they are actually thinking in any way similarly to what you or I do in writing these comments. The fact that we don't have a perfect, rigorous definition, and tend to rely on 'I know it when I see it,' does not mean LLMs do have it or that it will be trivial to get to them. All that is to say that when you say: > I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it. "Knowing for sure" is not required. A reasonable suspicion one way or the other based on experience is a good place to start. I also identified two specific things LLMs don't do - they are not self-motivated or goal-directed without prompting, and there is no evidence they possess a sense of self, even with the challenge of lack of definition that we face. | | |
| ▲ | nearbuy 3 days ago | parent | next [-] | | > But doesn't this rely on the same thing you suggest we don't have, which is a working and definable definition of consciousness? No, it's like saying we have no indication that humans have psychic powers and can levitate objects with their minds. The commenter is saying no human has ever demonstrated the ability to figure things out that aren't Turing computable and we have no reason to suspect this ability is even theoretically possible (for anything, human or otherwise). | |
| ▲ | vidarh 3 days ago | parent | prev [-] | | No, it rests on computability, Turing equivalence, and the total absence of both any kind of evidence to suggest we can exceed the Turing computable, and the lack of even a theoretical framework for what that would mean. Without that any limitations borne out of what LLMs don't currently do are irrelevant. | | |
| ▲ | ivraatiems 3 days ago | parent [-] | | That doesn't seem right to me. If I understand it right, your logic is: 1. Humans intellect is Turing computable.
2. LLMs are based on Turing-complete technology.
3. Therefore, LLMs can eventually equal human intellect. But if that is the right chain of assumptions, there's lots of issues with it. First, whether LLMs are Turing complete is a topic of debate. There are points for[0] and against[1]. I suspect they probably _are_, but that doesn't mean LLMs are tautologically indistinguishable from human intelligence. Every computer that uses a Turing-complete programming language can theoretically solve any Turing-computable problem. That does not mean they will ever be able to efficiently or effectively do so in real time under real constraints, or that they are doing so now in a reasonable amount real-world time using extant amounts of real-world computing power. The processor I'm using to write this might be able to perform all the computations needed for human intellect, but even if it could, that doesn't mean it can do it quickly enough to compute even a single nanosecond of actual human thought before the heat-death of the universe, or even the end of this century. So when you say: > Without that any limitations borne out of what LLMs don't currently do are irrelevant. It seems to me exactly the opposite is true. If we want technology that is anything approaching human intelligence, we need to find approaches which will solve for a number of things LLMs don't currently do. The fact that we don't know exactly what those things are yet is not evidence that those things don't exist. Not only do they likely exist, but the more time we spend simply scaling LLMs instead of trying to find them, the farther we are from any sort of genuine general intelligence. [0] https://arxiv.org/abs/2411.01992
[1] https://medium.com/heyjobs-tech/turing-completeness-of-llms-... |
|
| |
| ▲ | prmph 3 days ago | parent | prev | next [-] | | > I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it. Then why make an argument based on what you do not know? | | |
| ▲ | vidarh 3 days ago | parent [-] | | My point exactly. The person I replied to did just that. | | |
| ▲ | ivraatiems 3 days ago | parent [-] | | I think the parent is trying to point out the difference between our positions: You say the limits of LLMs don't matter, because we don't have definitions strong enough to describe them. I say the limits of LLMs do matter and the fact that we can't yet define them rigorously means we aren't able to fix them (assuming we want to). |
|
| |
| ▲ | nprateem 3 days ago | parent | prev [-] | | Anyone who believes an algorithm could be conscious needs to take mushrooms. | | |
| ▲ | visarga 3 days ago | parent | next [-] | | Consider the river metaphor: water carves the banks, banks channel the water. At any moment water and banks have the same shape. Model/algorithm is the banks. Water could be the experiences. Maybe the algorithm does not have consciousness, but it is part of it. They co-create each other. They are part of a recursive loop which cannot be explained statically, or part by part in isolation. | |
| ▲ | levitatorius 3 days ago | parent | prev | next [-] | | Yes! If algorithm is conscious (without being alive) then the eaten magic mushroom is also very conscious, judged by it's effect on the subject. | |
| ▲ | vidarh 3 days ago | parent | prev [-] | | Unless you can show me you can exceed the Turing computable, there is no reason to consider you any more than an algorithm. | | |
|
|
|
| ▲ | FloorEgg 3 days ago | parent | prev | next [-] |
| I think LLMs are conscious just in a very limited way. I think consciousness is tightly coupled to intelligence. If I had to guess, the current leading LLMs consciousness is most comparable to a small fish, with a conscious lifespan of a few seconds to a few minutes. Instead of perceiving water, nutrient gradients, light, heat, etc. it's perceiving tokens. It's conscious, but it's consciousness is so foreign to us it doesn't seem like consciousness. In the same way to an amoeba is conscious or a blade of grass is conscious but very different kind than we experience. I suspect LLMs are a new type of consciousness that's probably more different from ours than most if not all known forms of life. I suspect the biggest change that would bring LLM consciousness closer to us would be some for of continuous learning/model updating. Until then, even with RAG, and other clever teghniques I consider these models as having this really foreign slices of consciousness where they "feel" tokens and "act" out tokens, and they have perception, but their perception of the tokens is nothing like ours. If one looks closely at simple organisms with simple sensory organs and nervous systems its hard not to see some parallels. It's just that the shape of consciousness is extremely different than any life form. (perception bandwidth, ability to act, temporality, etc) Karl friston free energy principle gives a really interesting perspective on this I think. |
| |
| ▲ | wry_discontent 3 days ago | parent | next [-] | | What makes you think consciousness is tightly coupled to intelligence? | | |
| ▲ | XorNot 3 days ago | parent | next [-] | | It's hardly an unreasonable supposition: the one definitely conscious entities we know of are also the apex intelligence of the planet. To put it another way: lots of things are conscious, but humans are definitely the most conscious beings on Earth. | | |
| ▲ | wry_discontent an hour ago | parent | next [-] | | But that's not an answer. Why should intelligence and not some other quality be coupled to consciousness? In my experience, consciousness (by which I'm specifically talking about qualia/experience/awarenesss) doesn't at all seem tightly coupled to intelligence. Certainly not in a way that seems obvious to me. | |
| ▲ | CuriouslyC 3 days ago | parent | prev [-] | | I can understand what less cognizant or self aware means, but "less conscious" is confusing. What are you implying here? Are their qualia lower resolution? | | |
| ▲ | FloorEgg 3 days ago | parent | next [-] | | In a sense, yes. If one is to quantify consciousness it would probably make sense to think of it as an area of awareness and cognizance across time. Awareness scales with sensory scale and resolution (sensory receptors vs input token limits and token resolution). E.g. 128k tokens and tokens too coarse to count rs in strawberry. Cognizance scales with internal representations of awareness (probably some relation to vector space resolution and granularity, though I suspect there is more to it than just vector space) And the third component is time, how long the agent is conscious for. So something like... Time * awareness (receptors) * internal representations (cell diversity * # cells * connection diversity * # connections) There is no way this equation is right but I suspect it's sort of directionally correct. I'm deep in the subject but just riffing here, so take this with a lot of salt. | |
| ▲ | inglor_cz 3 days ago | parent | prev | next [-] | | Humans can reason why they are angry, for example. (At least some humans.) I am not sure if chimps can do the same. | |
| ▲ | noirscape 2 days ago | parent | prev [-] | | Pretty much. Most animals are both smarter than you expect, but also tend to be more limited in what they can reason about. It's why anyone who's ever taken care of a needy pet will inevitably reach the comparison that taking care of a pet is similar to taking care of a very young child; it's needy, it experiences emotions but it can't quite figure out on its own how to adapt to an environment besides what it grew up around/it's own instincts. They experience some sort of qualia (a lot of animals are pretty family-minded), but good luck teaching a monkey to read. The closest we've gotten is teaching them that if they press the right button, they get food, but they take basically their entire lifespan to understand a couple hundred words, while humans easily surpass that. IIRC some of the smartest animals in the world are actually rats. They experience a qualia very close to humans to the point that psychology experiments are often easily observable in rats. |
|
| |
| ▲ | FloorEgg 3 days ago | parent | prev [-] | | Karl Friston's free energy principle is probably roughly 80% of my reasons to think they're coupled. The rest comes from studying integrated information theories, architecture of brains and nervous systems and neutral nets, more broadly information theory, and a long tail of other scientific concepts (particle physics, chemistry, biology, evolution, emergence, etc...) | | |
| ▲ | wry_discontent an hour ago | parent [-] | | Isn't that begging the question? If you just accept the presupposition that intelligence is tightly coupled to consciousness, then all that makes perfect sense to me. But I don't see why I should accept that. It isn't obvious to me, and it doesn't match my own experience of being conscious. Totally possible that we're talking past each other. |
|
| |
| ▲ | procaryote 3 days ago | parent | prev [-] | | > I think LLMs are conscious just in a very limited way. I think consciousness is tightly coupled to intelligence. Why? | | |
| ▲ | FloorEgg 3 days ago | parent [-] | | I already answered under the other comment asking me why and if your curious I suggest looking for it. Very short answer is Karl Friston's free energy pricniple | | |
| ▲ | procaryote 2 days ago | parent [-] | | LLMs work nothing like Karl Friston's free energy principle though | | |
| ▲ | FloorEgg 2 days ago | parent [-] | | LLMs embody the free-energy principle computationally. They maintain an internal generative model of language and continually minimize “surprise”, the difference between predicted and actual tokens, during both training and infeence. In Friston’s terms, their parameters encode beliefs about the causes of linguistic input; forward passes generate predictions, and backpropagation adjusts internal states to reduce prediction error, just as perception updates beliefs to minimize free energy. During inference, autoregressive generation can be viewed as active inference: each new token selection aims to bring predicted sensory input (the next word) into alignment with the model’s expectations. In a broader sense, LLMs exemplify how a self-organizing system stabilizes itself in a high-dimensional environment by constantly reducing uncertainty about its inputs, a synthetic analogue of biological systems minimizing free energy to preserve their structural and informational coherence. | | |
| ▲ | procaryote 2 days ago | parent [-] | | You might have lost me but what you're describing doesn't sound like an LLM. E.g: > each new token selection aims to bring predicted sensory input (the next word) into alignment with the model’s expectations. what does that mean? An llm generates the next word based on what best matches its training, with some level of randomisation. Then it does it all again. It's not a percepual process trying to infer a reality from sensor data or anything | | |
| ▲ | FloorEgg 2 days ago | parent [-] | | > An llm generates the next word based on what best matches its training, with some level of randomisation. This is sort of accurate, but not precise. An LLM generates the next token by sampling from a probability distribution over possible tokens, where those probabilities are computed from patterns learned during training on large text datasets. The difference in our explanations is that you are biasing towards LLMs being fancy database indexes, and I am emphasizing that LLMs build a model of what they are trained on and respond based on that model, which is more like how brains and cells work than you are recognizing. (though I admit my understanding of microbiology places me just barely past peak Mt Stupid [Dunning Kruger], I don't really understand how individual cells do this and can only hand-wavey explain it). Both systems take input, pass it through a network of neurons, and produce output. Both systems are trying to minimize surprise in predictions. The differences are primarily in scale and complexity. Human brains have more types of neurons (units) and more types of connections (parameters). LLMs more closely mimic the prefrontal cortex, whereas e.g. the brainstem is a lot more different in terms of structure and cellular diversity. You can make a subjective ontological choice to draw categorical boundaries between them, or you can plot them on a continuum of complexity and scale. Personally I think both framings are useful, and to exclude either is to exclude part of the truth. My point is that if you draw a subjective categorical boundary around what you deem is consciousness and say that LLMs are outside of that, that is subjectively valid. You can also say that consciousness is a continuum, and individual cells, blades of grass, ants, mice, and people experience different types of consciousness on that continuum. If you take the continuum view, then what follows is a reasonable assumption that LLMs experience a very different kind of consciousness that takes in inputs at about the same rate as a small fish, models those inputs for a few seconds, and then produces outputs. What exactly that "feels" like is as foreign to me as it would be to you. I assume its even more foreign than what it would "feel" like to be a blade of grass. | | |
| ▲ | procaryote 2 days ago | parent [-] | | I'm not sure why you'd describe "sampling from a probability distribution over possible tokens" as "minimize surprise in predictions" other than to make it sound similar to the free energy thing. The free energy thing as I understand it has internal state, makes predictions, evaluates against new input and adjusts it internal state to continuously learn to predict new input better. This might if you squint look similar to training a neural network, although the mechanisms are different, but it's very distinct from the inference step | | |
| ▲ | FloorEgg a day ago | parent [-] | | "Minimize surprise" and "maximize accurate predictions" are the same thing mathematically. Minimize free energy = minimize prediction error. LLMs do everything modelled in the free energy principle, they just don't do continuous learning. (They don't do perceptual inference after RL) Your tone ("free energy thing" and "if you squint") comes off as dismissive and not intellectually honest. Here I thought you were actually curious, but I guess not? | | |
| ▲ | procaryote a day ago | parent [-] | | Poor wording on my side, I'm sorry. Thank you for explaining your reasoning | | |
|
|
|
|
|
|
|
|
|
|
| ▲ | thomastjeffery 3 days ago | parent | prev | next [-] |
| I think the most descriptive title I could give an LLM is "bias". An LLM is not "biased", it is bias; or at the very least, it's a good imitation of the system of human thinking/perception that we call bias. An LLM is a noise generator. It generates tokens without logic, arithmetic, or any "reason" whatsoever. The noise that an LLM generates is not truly random. Instead, the LLM is biased to generate familiar noise. The LLM itself is nothing more than a model of token familiarity. Nothing about that model can tell you why some tokens are more familiar with others, just like an accounting spreadsheet can't tell you why it contains a list of charges and a summation next to the word "total". It could just as easily contain the same kind of data with an entirely different purpose. What an LLM models is written human text. Should we really expect to not be surprised by the power and versatility of human-written text? --- It's clear that these statistical models are very good at thoughtless tasks, like perception and hallucination. It's also clear that they are very bad at thoughtful tasks like logic and arithmetic - the things that traditional software is made of. What no one has really managed to figure out is how to bridge that gap. |
| |
| ▲ | esafak 3 days ago | parent [-] | | LLMs today are great coders. Most humans are worse. | | |
| ▲ | inglor_cz 3 days ago | parent [-] | | LLMs ingested a lot of high-quality code during their training, plus LLMs being capable of programming is a huge commercial use case, so no wonder that they are good at coding. My experience, though, is that they aren't good at defining the task to be coded, or thinking about some unexpected side-effects. Code that will be left for them to develop freely will likely become bloated quite fast. |
|
|
|
| ▲ | spragl 2 days ago | parent | prev | next [-] |
| This is how I see LLMs as well. The main problem with the article is that it is meandering around in ill-conceived concepts, like thinking, smart, intelligence, understanding... Even AI. What they mean to the author is not what they mean to me, and still different to they mean to the other readers. There are all these comments from different people throughout the article, all having their own thoughts on those concepts. No wonder it all seem so confusing. It will be interesting when the dust settles, and a clear picture of LLMs can emerge that all can agree upon. Maybe it can even help us define some of those ill-defined concepts. |
| |
| ▲ | ojosilva 2 days ago | parent [-] | | I think the consensus in the future will be that LLMs were, after all, stochastic parrots. The difference with what we think today is that in the future we'll have a new definition of stochastic parrots, a recognition that stochastic parrots can actually be very convincing and extremely useful, and that they exhibit intelligence-like capabilities that seemed unattainable by any technology up to that point, but LLMs were not a "way forward" for attaining AGI. They will plateau as far as AGI metrics go. These metrics keep advancing to stay ahead of LLM, like a Achilles and the Turtle. But LLMs will keep improving as tooling around it becomes more sophisticated and integrated, and architecture evolves. |
|
|
| ▲ | heresie-dabord 3 days ago | parent | prev | next [-] |
| > a midpoint between "AIs are useless and do not actually think" and "AIs think like humans" LLMs (AIs) are not useless. But they do not actually think. What is trivially true is that they do not actually need to think. (As far as the Turing Test, Eliza patients, and VC investors are concerned, the point has been proven.) If the technology is helping us write text and code, it is by definition useful. > In 2003, the machine-learning researcher Eric B. Baum published a book called “What Is Thought?” [...] The gist of Baum’s argument is that understanding is compression, and compression is understanding. This is incomplete. Compression is optimisation, optimisation may resemble understanding, but understanding is being able to verify that a proposition (compressed rule or assertion) is true or false or even computable. > —but, in my view, this is the very reason these models have become increasingly intelligent. They have not become more intelligent. The training process may improve, the vetting of the data improved, the performance may improve, but the resemblance to understanding only occurs when the answers are provably correct. In this sense, these tools work in support of (are therefore part of) human thinking. The Stochastic Parrot is not dead, it's just making you think it is pining for the fjords. |
| |
| ▲ | crazygringo 3 days ago | parent [-] | | > But they do not actually think. I'm so baffled when I see this being blindly asserted. With the reasoning models, you can literally watch their thought process. You can see them pattern-match to determine a strategy to attack a problem, go through it piece-by-piece, revisit assumptions, reformulate strategy, and then consolidate findings to produce a final result. If that's not thinking, I literally don't know what is. It's the same process I watch my own brain use to figure something out. So I have to ask you: when you claim they don't think -- what are you basing this on? What, for you, is involved in thinking that the kind of process I've just described is missing? Because I genuinely don't know what needs to be added here for it to become "thinking". | | |
| ▲ | Terr_ 3 days ago | parent | next [-] | | > I'm so baffled when I see this being blindly asserted. With the reasoning models, you can literally watch their thought process. Not true, you are falling for a very classic (prehistoric, even) human illusion known as experiencing a story: 1. There is a story-like document being extruded out of a machine humans explicitly designed for generating documents, and which humans trained on a bajillion stories humans already made. 2. When you "talk" to a chatbot, that is an iterative build of a (remote, hidden) story document, where one of the characters is adopting your text-input and the other's dialogue is being "performed" at you. 3. The "reasoning" in newer versions is just the "internal monologue" of a film noir detective character, and equally as fictional as anything that character "says out loud" to the (fictional) smokin-hot client who sashayed the (fictional) rent-overdue office bearing your (real) query on its (fictional) lips. > If that's not thinking, I literally don't know what is. All sorts of algorithms can achieve useful outcomes with "that made sense to me" flows, but that doesn't mean we automatically consider them to be capital-T Thinking. > So I have to ask you: when you claim they don't think -- what are you basing this on? Consider the following document from an unknown source, and the "chain of reasoning" and "thinking" that your human brain perceives when encountering it: My name is Robot Robbie.
That high-carbon steel gear looks delicious.
Too much carbon is bad, but that isn't true here.
I must ask before taking.
"Give me the gear, please."
Now I have the gear.
It would be even better with fresh manure.
Now to find a cow, because cows make manure.
Now whose reasoning/thinking is going on? Can you point to the mind that enjoys steel and manure? Is it in the room with us right now? :PIn other words, the reasoning is illusory. Even if we accept that the unknown author is a thinking intelligence for the sake of argument... it doesn't tell you what the author's thinking. | | |
| ▲ | crazygringo 3 days ago | parent [-] | | You're claiming that the thinking is just a fictional story intended to look like it. But this is false, because the thinking exhibits cause and effect and a lot of good reasoning. If you change the inputs, the thinking continues to be pretty good with the new inputs. It's not a story, it's not fictional, it's producing genuinely reasonable conclusions around data it hasn't seen before. So how is it therefore not actual thinking? And I have no idea what your short document example has to do with anything. It seems nonsensical and bears no resemblance to the actual, grounded chain of thought processes high-quality reasoning LLM's produce. > OK, so that document technically has a "chain of thought" and "reasoning"... But whose? What does it matter? If an LLM produces output, we say it's the LLM's. But I fail to see how that is significant? | | |
| ▲ | czl 3 days ago | parent | next [-] | | > So how is it therefore not actual thinking? Many consider "thinking" something only animals can do, and they are uncomfortable with the idea that animals are biological machines or that life, consciousness, and thinking are fundamentally machine processes. When an LLM generates chain-of-thought tokens, what we might casually call “thinking,” it fills its context window with a sequence of tokens that improves its ability to answer correctly. This “thinking” process is not rigid deduction like in a symbolic rule system; it is more like an associative walk through a high-dimensional manifold shaped by training. The walk is partly stochastic (depending on temperature, sampling strategy, and similar factors) yet remarkably robust. Even when you manually introduce logical errors into a chain-of-thought trace, the model’s overall accuracy usually remains better than if it had produced no reasoning tokens at all. Unlike a strict forward- or backward-chaining proof system, the LLM’s reasoning relies on statistical association rather than brittle rule-following. In a way, that fuzziness is its strength because it generalizes instead of collapsing under contradiction. | | |
| ▲ | Terr_ 3 days ago | parent [-] | | Well put, and if it doesn't notice/collapse under introduced contradictions, that's evidence it's not the kind of reasoning we were hoping for. The "real thing" is actually brittle when you do it right. | | |
| ▲ | czl 3 days ago | parent [-] | | Human reasoning is, in practice, much closer to statistical association than to brittle rule-following. The kind of strict, formal deduction we teach in logic courses is a special, slow mode we invoke mainly when we’re trying to check or communicate something, not the default way our minds actually operate. Everyday reasoning is full of heuristics, analogies, and pattern matches: we jump to conclusions, then backfill justification afterward. Psychologists call this “post hoc rationalization,” and there’s plenty of evidence that people form beliefs first and then search for logical scaffolding to support them. In fact, that’s how we manage to think fluidly at all; the world is too noisy and underspecified for purely deductive inference to function outside of controlled systems. Even mathematicians, our best examples of deliberate, formal thinkers, often work this way. Many major proofs have been discovered intuitively and later found to contain errors that didn’t actually invalidate the final result. The insight was right, even if the intermediate steps were shaky. When the details get repaired, the overall structure stands. That’s very much like an LLM producing a chain of reasoning tokens that might include small logical missteps yet still landing on the correct conclusion: the “thinking” process is not literal step-by-step deduction, but a guided traversal through a manifold of associations shaped by prior experience (or training data, in the model’s case). So if an LLM doesn’t collapse under contradictions, that’s not necessarily a bug; it may reflect the same resilience we see in human reasoning. Our minds aren’t brittle theorem provers; they’re pattern-recognition engines that trade strict logical consistency for generalization and robustness. In that sense, the fuzziness is the strength. | | |
| ▲ | Terr_ 2 days ago | parent [-] | | > The kind of strict, formal deduction we teach in logic courses is a special, slow mode Yes, but that seems like moving the goalposts. The stricter blends of reasoning are what everybody is so desperate to evoke from LLMs, preferably along with inhuman consistency, endurance, and speed. Just imagine the repercussions if a slam-dunk paper came out tomorrow, which somehow proved the architectures and investments everyone is using for LLMs are a dead-end for that capability. | | |
| ▲ | crazygringo 2 days ago | parent | next [-] | | > The stricter blends of reasoning are what everybody is so desperate to evoke from LLMs This is definitely not true for me. My prompts frequently contain instructions that aren't 100% perfectly clear, suggest what I want rather than formally specifying it, typos, mistakes, etc. The fact that the LLM usually figures out what I meant to say, like a human would, is a feature for me. I don't want an LLM to act like an automated theorem prover. We already have those. Their strictness makes them extremely difficult to use, so their application is extremely limited. | |
| ▲ | czl 2 days ago | parent | prev [-] | | I get the worry. AFAIK most of the current capex is going into scalable parallel compute, memory, and networking. That stack is pretty model agnostic, similar to how all that dot com fiber was not tied to one protocol. If transformers stall, the hardware is still useful for whatever comes next. On reasoning, I see LLMs and classic algorithms as complements. LLMs do robust manifold following and associative inference. Traditional programs do brittle rule following with guarantees. The promising path looks like a synthesis where models use tools, call code, and drive search and planning methods such as MCTS, the way AlphaGo did. Think agentic systems that can read, write, execute, and verify. LLMs are strongest where the problem is language. Language co evolved with cognition as a way to model the world, not just to chat. We already use languages to describe circuits, specify algorithms, and even generate other languages. That makes LLMs very handy for specification, coordination, and explanation. LLMs can also statistically simulate algorithms, which is useful for having them think about these algorithms. But when you actually need the algorithm, it is most efficient to run the real thing in software or on purpose built hardware. Let the model write the code, compose the tools, and verify the output, rather than pretending to be a CPU. To me the risk is not that LLMs are a dead end, but that people who do not understand them have unreasonable expectations. Real progress looks like building systems that use language to invent and implement better tools and route work to the right place. If a paper lands tomorrow that shows pure next token prediction is not enough for formal reasoning, that would be an example of misunderstanding LLMs, not a stop sign. We already saw something similar when Minsky and Papert highlighted that single layer perceptrons could not represent XOR, and the field later moved past that with multilayer networks. Hopefully we remember that and learn the right lesson this time. |
|
|
|
| |
| ▲ | rustystump 3 days ago | parent | prev [-] | | The problem is that the overwhelming majority of input it has in-fact seen somewhere in the corpus it was trained on. Certainly not one for one but easily an 98% match. This is the whole point of what the other person is trying to comment on i think. The reality is most of language is regurgitating 99% to communicate an internal state in a very compressed form. That 1% tho maybe is the magic that makes us human. We create net new information unseen in the corpus. | | |
| ▲ | crazygringo 3 days ago | parent | next [-] | | > the overwhelming majority of input it has in-fact seen somewhere in the corpus it was trained on. But it thinks just great on stuff it wasn't trained on. I give it code I wrote that is not in its training data, using new concepts I've come up with in an academic paper I'm writing, and ask it to extend the code in a certain way in accordance with those concepts, and it does a great job. This isn't regurgitation. Even if a lot of LLM usage is, the whole point is that it does fantastically with stuff that is brand new too. It's genuinely creating new, valuable stuff it's never seen before. Assembling it in ways that require thinking. | | |
| ▲ | rustystump 3 days ago | parent | next [-] | | I think you may think too highly of academic papers or more so that they oft still only have 1% in there. | | |
| ▲ | crazygringo 3 days ago | parent [-] | | I think you're missing the point. This is my own paper and these are my own new concepts. It doesn't matter if the definition of the new concepts are only 1% of the paper, the point is they are the concepts I'm asking the LLM to use, and are not in its training data. | | |
| ▲ | Terr_ 3 days ago | parent [-] | | How would one prove the premise that a concept is not present in the training data? With how much data is being shoveled in there, our default assumption should be that significant components are present. | | |
| ▲ | crazygringo 2 days ago | parent [-] | | That would be a weird default assumption. It's not hard to come up with new ideas. In fact, it's trivial. And if you want to know if a specific concept is known by the LLM, you can literally ask it. It generally does a great job of telling you what it is and is not familiar with. |
|
|
| |
| ▲ | zeroonetwothree 3 days ago | parent | prev [-] | | I think it would be hard to prove that it's truly so novel that nothing similar is present in the training data. I've certainly seen in research that it's quite easy to miss related work even with extensive searching. |
| |
| ▲ | the_pwner224 3 days ago | parent | prev [-] | | Except it's more than capable of solving novel problems that aren't in the training set and aren't a close match to anything in the training set. I've done it multiple times across multiple domains. Creating complex Excel spreadsheet structures comes to mind, I just did that earlier today - and with plain GPT-5, not even -Thinking. Sure, maybe the Excel formulas themselves are a "98% match" to training data, but it takes real cognition (or whatever you want to call it) to figure out which ones to use and how to use them appropriately for a given situation, and how to structure the spreadsheet etc. | | |
| ▲ | rustystump 3 days ago | parent [-] | | I think people confuse novel to them with novel to humanity. Most of our work is not so special | | |
|
|
|
| |
| ▲ | baq 3 days ago | parent | prev [-] | | Brains are pretrained models, change my mind. (Not LLMs obviously, to be perfectly clear) | | |
| ▲ | hamdingers 3 days ago | parent | next [-] | | Brains continue learning from everything they do for as long as they're in use. Pretrained models are static after initial training. | |
| ▲ | zeroonetwothree 3 days ago | parent | prev [-] | | If you are right, then I certainly cannot change your mind. | | |
| ▲ | baq 2 days ago | parent [-] | | Show a snake to a 1yo and explain how the kid’s reaction is not pretrained. It’s called instinct in biology, but the idea is the same. |
|
|
|
|
|
| ▲ | stickfigure 3 days ago | parent | prev | next [-] |
| > Turing Test IMO none of the current crop of LLMs truly pass the Turing Test. If you limit the conversation to an hour or two, sure - but if you let a conversation run months or years I think it will be pretty easy to pick the machine. The lack of continuous learning and the quality dropoff as the context window fills up will be the giveaways. |
|
| ▲ | shadyKeystrokes 3 days ago | parent | prev [-] |
| By that reasoning all that is missing is what a human brings as "stimuli" to review, refine and reevaluate as complete. |
| |
| ▲ | ivraatiems 3 days ago | parent [-] | | I don't think that's quite the only thing missing, I also discussed the idea of a sense of self. But even if that was all there was, it's a pretty big "but". |
|
