Can’t agree with you here.

> I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.

In love how people get salty about people not going along with a superficial supposition just because they can’t definitively prove it wrong.

> All that while still not knowing how either kind actually works.

We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. We do know exactly how each part of an LLM works even if the combined behavior is too cryptic to feasibly analyze at the moment. We do not understand all of the functions of an actual neuron. Openworm isn’t even close to accurately simulating the 302 neurons of a roundworm and you’d need over 200 million roundworms working in conjunction to equal the number of neurons in one human brain.

My dog seems convinced that the malevolent invader in a mailman uniform would break in and attack us if she didn’t fiercely bark at him, six days per week. I certainly can’t prove the mailman doesn’t want to kill us, and that the mailman wasn’t solely deterred by her barking. Empirically, the mailman goes away soon after she starts barking, and we’ve sustained zero mailman assaults after hundreds of purported attempts. Maybe I should just run with it? Her model is too simple to come up with the obviously correct answer, but it’s not even directionally accurate.

The burden of proof is on the person making the claim, which in this case, is that these comparatively simple logical constructs are remotely comparable to the complexity of biological systems.

> ... are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.

If you're claiming that the training objective tells us what kind of internal mechanisms the training produced, then I think that's just plain wrong.

Next-token prediction describes the optimization target, not the internal mechanisms that the training produced.

In the same way for the natural evolution of humans, DNA replication is the evolutionary objective. It's not a description of the internal mechanisms that evolution has produced.

As an example, we know that neural networks can be trained to develop generalized algorithms for arithmetic.

They might first memorize the training examples, then with further training transition to a solution that generalizes correctly to unseen examples.

In some cases we've even reverse-engineered the evolved internal mechanisms and found structured arithmetic algorithms rather than rote memorization. Interestingly, for modular addition this can involve Fourier representations, which isn't an algorithm I would have guessed gradient descent training of neural networks would produce.

Disagree about the burden of proof. We have no better model for how human decision making works than LLMs. Humans are constantly predicting the next moment. We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.

<< We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database.

Oh man, how much did you read on tip of the tongue?