> Modern coding AI models are not just probability crunching transformers. (...) The complete package includes things like highly curated training data, specialized tokenizers, pre and post training regimens, guardrails, optimized system prompts etc, all tuned to coding.

It seems you were not aware you ended up describing probabilistic coding transformers. Each and every single one of those details are nothing more than strategies to apply constraints to the probability distributions used by the probability crunching transformers. I mean, read what you wrote: what do you think that "curated training data" means?

> Put it all together and you get one shot performance on generating the type of code that was unthinkable even a year ago.

This bit here says absolutely nothing.

>The complete package includes things like highly curated training data, specialized tokenizers, pre and post training regimens, guardrails, optimized system prompts etc, all tuned to coding.

And even with all that, they still produce garbage way too often. If we continue the "car" analogy, the car would crash randomly sometimes when you leave the driveway, and sometimes it would just drive into the house. So you add all kinds of fancy bumpers to the car and guard rails to the roads, and the car still runs off the road way too often.

Except we should aim to reduce the boilerplate through good design, instead of creating more of it on an industrial scale.

>In current coding models the transformer bit is just one part of what is really an expert system. The complete package includes things like highly curated training data, specialized tokenizers, pre and post training regimens, guardrails, optimized system prompts etc, all tuned to coding. Put it all together and you get one shot performance on generating the type of code that was unthinkable even a year ago.

This is lipstick on a pig. All those methods are impressive, but ultimately workarounds for an idea that is fundamentally unsuitable for programming.

>That's >90% of all code in the wild. Probably more.

Maybe, but not 90% of time spent on programming. Boilerplate is easy. It's the 20%/80% rule in action.

I don't deny these tools can be useful and save time - but they can't be left to their own devices. They need to be tightly controlled and given narrow scopes, with heavy oversight by an SME who knows what the code is supposed to be doing. "Design W module with X interface designed to do Y in Z way", keeping it as small as possible and reviewing it to hell and back. And keeping it accountable by making tests yourself. Never let it test itself, it simply cannot be trusted to do so.

LLMs are incredibly good at writing something that looks reasonable, but is complete nonsense. That's horrible from a code maintenance perspective.

>hallucinations

This is the first word that came to mind when reading the comment above yours. Like:

>They can't, despite marketing, really reason

They aren't, despite marketing, really hallucinations.

Now I understand why these companies don't want to market using terms like "extrapolated bullshit", but I don't understand how there is any technological solution to it without starting from a fresh base.

> Not to disagree, but "non-english" isn't exactly relevant.

how so? programs might use english words but are decidedly not english.

Does speaking in your native language, speaking in a second language, thinking about your life and doing maths feel exactly the same to you?

> Prove to me that human thought is not predicting the most probable next token.

Explain the concept of color to a completely blind person. If their brain does nothing but process tokens this should be easy.

> How can you tell a human actually understands?

What a strange question coming from a human. I would say if you are a human with a consciousness you are able to answer this for yourself, and if you aren't no answer will help.

That is ill-posed. Take any algorithm at all, e.g. a TSP solver. Make a "most probable next token predictor" that takes the given traveling salesman problem, runs the solver, and emits the first token of the solution, then reruns the solver and emits the next token, and so on.

By this thought experiment you can make any computational process into "predict the most probable next token" - at an extreme runtime cost. But if you do so, you arguably empty the concept "token predictor" of most of its meaning. So you would need to more accurately specify what you mean by a token predictor so that the answer isn't trivially true (for every kind of thought that's computation-like).

Just a couple of weeks ago on mid-range models. The problem is not implementation or refinement - the core idea is fundamentally flawed.

>from a neuroscience perspective, there's a growing consensus among neuroscientists that the brain is fundamentally a token predictor, and that it works on exactly the same principles as LLMs

Hahahahahaha.

Oh god, you're serious.

Sure, let's just completely ignore all the other types of processing that the brain does. Sensory input processing, emotional regulation, social behavior, spatial reasoning, long and short term planning, the complex communication and feedback between every part of the body - even down to the gut microbiome.

The brain (human or otherwise) is incredibly complex and we've barely scraped the surface of how it works. It's not just nuerons (which are themselves complex), it's interactions between thousands of types of cells performing multiple functions each. It will likely be hundreds of years before we get a full grasp on how it truly works - if we ever do at all.

> The only difference between a brain and an LLM maybe the size of its memory, and what kind and quality of data it's trained on.

This is trivially proven false, because LLMs have far larger memory than your average human brain and are trained on far more data. Yet they do not come even close to approximating human cognition.

> and that from a neuroscience perspective, there's a growing consensus among neuroscientists that the brain is fundamentally a token predictor, and that it works on exactly the same principles as LLMs

Can you cite at least one recognized, credible neuroscientist who makes this claim?

Look you don't have to lie at every opportunity you get. You are fully aware and know what you've written is bullshit.

Tokens are a highly specific transformer exclusive concept. The human brain doesn't run a byte pair encoding (BPE) tokenizer [0] in their head. anything as tokens. It uses asynchronous time varying spiking analog signals. Humans are the inventors of human languages and are not bound to any static token encoding scheme, so this view of what humans do as "token prediction" requires either a gross misrepresentation of what a token is or what humans do.

If I had to argue that humans are similar to anything in machine learning research specifically, I would have to argue that they extremely loosely follow the following principles:

* reinforcement learning with the non-brain parts defining the reward function (primarily hormones and pain receptors)

* an extremely complicated non-linear kalman filter that not only estimates the current state of the human body, but also "estimates" the parameters of a sensor fusing model

* there is a necessary projection of the sensor fused result that then serves as available data/input to the reinforcement learning part of the brain

Now here are two big reasons why the model I describe is a better fit:

The first reason is that I am extremely loose and vague. By playing word games I have weaseled myself out of any specific technology and am on the level of concepts.

The second reason is that the kalman filter concept here is general enough that it also includes predictor models, but the predictor model here is not the output that drives human action, because that would logically require the dataset to already contain human actions, which is what you did, you assume that all learning is imitation learning.

In my model, any internal predictor model that is part of the kalman filter is used to collect data, not drive human action. Actions like eating or drinking are instead driven by the state of the human body, e.g. hunger is controlled through leptin and insulin and others. All forms of work, no matter how much of a detour it represents, ultimately has the goal of feeding yourself or your family (=reproduction).

[0] A BPE tokenizer is a piece of human written software that was given a dataset to generate an efficient encoding scheme and the idea itself is completely independent of machine learning and neural networks. The fundamental idea behind BPE is that you generate a static compression dictionary and never change it.

You seem to be an LLM

Most adult humans don't have "true intelligence" so I don't quite get the point

> Every AI-related invention is hyped as "intelligence" but turns out to be "Necessary but Not Sufficient" for true intelligence.

Alternatively, the goalposts keep being moved.

Both that, and that we should not expect LLMs to achieve ability with humans as baseline comparison. It’s as if cars were rapidly getting better due to some new innovation, and expecting them to fly within a year. It’s a new, and different thing, where the universality of ”plausibly sounding” coherent text appeared to be general, when it’s advanced pattern matching. Nothing wrong with that, pattern matching is extremely useful, but drawing the equal sign to human cognition is extremely premature, and a bet that is very likely be wrong.

Alpha Zero is not trying to be AGI.

> The premise that an AI needs to do Y "as we do" to be good at X because humans use Y to be good at X needs closer examination.

I don't see it being used as a premise. It see it as speculation that is trying to understand why this type of AI underperforms at certain types of tasks. Y may not be necessary to do X well, but if a system is doing X poorly and the difference between that system and another system seems to be Y, it's worth exploring if adding Y would improve the performance.

> the developers and users fully embracing it are experiencing productivity boosts unseen previously

This is the kind of thing that I disagree with. Over the last 75 years we’ve seen enormous productivity gains.

You think that LLMs are a bigger productivity boost than moving from physically rewiring computers to using punch cards, from running programs as batch processes with printed output to getting immediate output, from programming in assembly to higher level languages, or even just moving from enterprise Java to Rails?

The studies I've seen for AI actually improving productivity are a lot more modest than what the hype would have you believe. For example: https://www.youtube.com/watch?v=tbDDYKRFjhk

Skepticism isn't the same thing as fighting the future.

I will call something AGI when it can reliably solve novel problems it hasn't been pre-trained on. That's my goal post and I haven't moved it.

!= is "not equal". The symbol for "not a subset of" is ⊄, which you will note, I did not use.

In cybernetics, this label has existed for a long time.

Unfortunately, discourse has followed an epistemic trajectory influenced by Hollywood and science fiction, making clear communication on the subject nearly impossible without substantial misunderstanding.

So when an LLM all-too-often produces garbage, can we then call it "Artificial Stupidity"?

> Anyone that says LLMs do not qualify as AI are the same people who will continue to move the goal posts for AGI.

I have the complete opposite feeling. The layman understanding of the term "AI" is AGI, a term that only needs to exist because researchers and businessmen hype their latest creations as AI.

The goalposts for AI don't move but the definition isn't precise but we know it when we see it.

AI, to the layman, is Skynet/Terminator, Asimov's robots, Data, etc.

The goalposts moving that you're seeing is when something the tech bubble calls AI escapes the tech bubble and everyone else looks at it and says, no, that's not AI.

The problem is that everything that comes out of the research efforts toward AI, the tech industry calls AI despite it not achieving that goal by the common understanding of the term. LLMs were/are a hopeful AI candidate but, as of today, they aren't but that doesn't stop OpenAI from trying to raise money using the term.

> The sooner people stop worrying about a label for what you feel fits LLMs best, the sooner they can find the things they (LLMs) absolutely excel at and improve their (the user's) workflows.

This is not a fault of the users. These labels are pushed primarily by "AI" companies in order to hype their products to be far more capable than they are, which in turn increases their financial valuation. Starting with "AI" itself, "superintelligence", "reasoning", "chain of thought", "mixture of experts", and a bunch of other labels that anthropomorphize and aggrandize their products. This is a grifting tactic old as time itself.

From Sam Altman[1]:

> We are past the event horizon; the takeoff has started. Humanity is close to building digital superintelligence

Apologists will say "they're just words that best describe these products", repeat Dijkstra's "submarines don't swim" quote, but all of this is missing the point. These words are used deliberately because of their association to human concepts, when in reality the way the products work is not even close to what those words mean. In fact, the fuzzier the word's definition ("intelligence", "reasoning", "thought"), the more valuable it is, since it makes the product sound mysterious and magical, and makes it easier to shake off critics. This is an absolutely insidious marketing tactic.

The sooner companies start promoting their products honestly, the sooner their products will actually benefit humanity. Until then, we'll keep drowning in disinformation, and reaping the consequences of an unregulated marketplace of grifters.

[1]: https://blog.samaltman.com/the-gentle-singularity

I’m not an expert on this, so I’m not familiar with what RDF graphs are, but I feel like everything you’re describing happens textually, and used as context? That is, it’s not at all ”learning” the way it’s learning during training, but by writing things down to refer to them later? As you say - ”ask it to summarize the insights for later use” - this is fundamentally different from the types of ”insights” it can have during training. So, it can take notes about your code and refer back to them, but it only has meaningful ”knowledge” about code it came across in training.

To me as a layman, this feels like a clear explanation of how these tools break down, why they start going in circles when you reach a certain complexity, why they make a mess of unusual requirements, and why they have such an incredible nuanced grasp of complex ideas that are widely publicized, while being unable to draw basic conclusions about specific constraints in your project.

Heck, replace "AI" with almost any noun and you can close your eyes to any and all criticism!

Agreed. I find LLMs incredibly useful for my work and I'm amazed at what they can do.

But I'm really worried that the benefits are very localized, and that the externalized costs are vast, and the damage and potential damage isn't being addressed. I think that they could be one of the greatest ever drivers of inequality as a privileged few profit at the expense of the many.

Any debates seem neglect this as they veer off into AGI Skynet fantasy land damage rather than grounded real world damage. This seems to be deliberate distraction.