Remix.run Logo
▲ spacedcowboy 3 hours ago

> I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI"

Me too, my first neural network research was for my PhD, 35 years or so ago, on a Meiko computing surface.

My bar for AGI over what we have (generative AI or genAI) is still fairly high, fortunately I do not believe that the current genAI path is set to lead us to AGI, there are very fundamental differences.

Right now for genAI, the model is static, the weights do not change, all “memory” is in the context, and even with the context sometimes exhibiting “J-space” (a shared "workspace" for deliberate, multi-step reasoning spontaneously created in the context memory), its ephemeral; it doesn't inform the weights in the base model. Even when we try to do “weight-space programming” to actually modify the model itself, the supervisor-model needed to do that is far and away larger than the one being operated on, and once a change has been made the supervisor-model is now out of sync with its original representation and it becomes less useful the more it diverges. We have not solved "thinking" today, we “just” have a high-probability path between states in a static weight-space which we follow and feed back.

What thinking requires is the ability to generate hypotheses (which requires internal state modification), to hold them and test them against other facts and hypotheses, and to update the internal state on test results, negative or positive. LLM’s don’t do that, they’re actually more like a database lookup, where the input tokens act as keys into a high-dimensional space where information is stored. You could even consider “the model” to be just an entropic compression of lots of data, and the inference step to be the lookup. It’s the scope, and the fuzziness of the lookup that makes them seem to work so well. Lots of results come back, and the best-path through those results wins. It’s not “thinking” as we understand it.

This is why humans are still a crucial component: we set the boundary conditions of these lookups, we set the starting point and the goals, and we adjust the success criteria to guide the chat - note that these are all the comprehension parts of the task. It’s also why models will hallucinate: the fundamental design does not lend itself to a null hypothesis. Even in the newest "reasoning" models, there is still no feedback of state back into the model itself. What they are effectively doing is running a Monte Carlo pass through the problem space—generating hidden branches and picking a likely path—before reporting back. Because the underlying weights never update from that process, it’s still just a sophisticated search, not genuine comprehension - there is still no “that makes no sense, reconfigure” step.

Stuart Kauffman (“The Origins Of Order”) would say its weights are stuck in the frozen realm of order, and that it takes a genuine phase transition to get to where the real action is - at the boundary of chaos and order where humans reside. Tweaking and fiddling with the system while it remains locked in the stasis of static weights is never going to effect that state change.

That’s not to say genAI is not useful, it’s freaking awesome, possibly the best invention in the history of time, but it doesn’t truly “think”, it maps.

▲lordnacho 6 minutes ago | parent | next [-]

What I find is that the LLM is like a smart person who knows when to use a tool. And it is freaking good at finding tools to use.

A bit like a person who is smart enough to figure out how an excavator works, or a drill, or sewing needle.

It doesn't feel like it ever internalizes anything, it just knows there's a million tools available, and the extent to which it thinks is that is can do enough to plan some tool uses, look at the result, and continue until some goal is reached.

When it comes to hypothesis testing, that seems to be what it does. It says "hmm, I'm going to open up my hypothesis testing kit (typically python) and drive this data through it". At least this is what I've seen when I put some statistics through it.

In general, it seems like the intelligence is mainly a kind of glue. It knows enough about what "right" looks like in each domain, but as a sort of consumer. He doesn't bake the cake, he buys it in.

▲kenjackson an hour ago | parent | prev | next [-]

It seems like your definition of AGI is too focused on implementation details. IMO it has to be focused on behavior. Once you say something like “the weights do not change” then I think we’ve lost the plot. Instead say this is what an intelligent being can do, but a GenAI cannot.

▲HarHarVeryFunny an hour ago | parent | next [-]

I agree that definitions of intelligence should be based on capability rather than implementation, but given that we're discussing LLMs where the intelligence is in the weights, then "the weights don't change" can obviously be taken as pointing out that the model can't learn - the best it can do is memorize and load memorized stuff into context.

Memorizing facts is not the same as learning skills. Implementation-wise memorization can just be RAG, but learning of skills does need to be in the "weights" - whatever is driving the actual behavior of the model.

▲imtringued an hour ago | parent | prev [-]

I don't get it. Weights don't change means the model can't learn.

This is not merely an implementation detail, continual learning is the core of AGI.

▲brainwad 23 minutes ago | parent [-]

It means the weights by themselves can't learn. But they could be used in a harness that can learn in other ways. Insisting that it be done at the neural net level seems anthropocentric - because we form memories via neural connections, that everything else must, too.

Also: I wouldn't say a human with amnesia who relied on notes to their future selves lacks intelligence, so I'm not sure being able to form memories is in fact necessary.

▲curt15 2 hours ago | parent | prev [-]

Self-modifying neural nets?