Even if interpretability of specific models or features within them is an open area of research, the mechanics of how LLMs work to produce results are observable and well-understood, and methods to understand their fundamental limitations are pretty solid these days as well.

Is there anything to be gained from following a line of reasoning that basically says LLMs are incomprehensible, full stop?

▲

famouswaffles 4 hours ago | parent | next [-]

>Even if interpretability of specific models or features within them is an open area of research, the mechanics of how LLMs work to produce results are observable and well-understood, and methods to understand their fundamental limitations are pretty solid these days as well.

If you train a transformer on (only) lots and lots of addition pairs, i.e '38393 + 79628 = 118021' and nothing else, the transformer will, during training discover an algorithm for addition and employ it in service of predicting the next token, which in this instance would be the sum of two numbers.

We know this because of tedious interpretability research, the very limited problem space and the fact we knew exactly what to look for.

Alright, let's leave addition aside (SOTA LLMs are after all trained on much more) and think about another question. Any other question at all. How about something like:

"Take a capital letter J and a right parenthesis, ). Take the parenthesis, rotate it counterclockwise 90 degrees, and put it on top of the J. What everyday object does that resemble?"

What algorithm does GPT or Gemini or whatever employ to answer this and similar questions correctly ? It's certainly not the one it learnt for addition. Do you Know ? No. Do the creators at Open AI or Google know ? Not at all. Can you or they find out right now ? Also No.

Let's revisit your statement.

"the mechanics of how LLMs work to produce results are observable and well-understood".

Observable, I'll give you that, but how on earth can you look at the above and sincerely call that 'well-understood' ?

▲

striking 3 hours ago | parent | next [-]

It's pattern matching, likely from typography texts and descriptions of umbrellas. My understanding is that the model can attempt some permutations in its thinking and eventually a permutation's tokens catch enough attention to attempt to solve, and that once it is attending to "everyday object", "arc", and "hook", it will reply with "umbrella".

Why am I confident that it's not actually doing spatial reasoning? At least in the case of Claude Opus 4.6, it also confidently replies "umbrella" even when you tell it to put the parenthesis under the J, with a handy diagram clearly proving itself wrong: https://claude.ai/share/497ad081-c73f-44d7-96db-cec33e6c0ae3 . Here's me specifically asking for the three key points above: https://claude.ai/share/b529f15b-0dfe-4662-9f18-97363f7971d1

I feel like I have a pretty good intuition of what's happening here based on my understanding of the underlying mathematical mechanics.

Edit: I poked at it a little longer and I was able to get some more specific matches to source material binding the concept of umbrellas being drawn using the letter J: https://claude.ai/share/f8bb90c3-b1a6-4d82-a8ba-2b8da769241e

▲

famouswaffles 2 hours ago | parent [-]

>It's pattern matching, likely from typography texts and descriptions of umbrellas.

"Pattern matching" is not an explanation of anything, nor does it answer the question I posed. You basically hand waved the problem away in conveniently vague and non-descriptive phrase. Do you think you could publish that in a paper for ext ?

>Why am I confident that it's not actually doing spatial reasoning? At least in the case of Claude Opus 4.6, it also confidently replies "umbrella" even when you tell it to put the parenthesis under the J, with a handy diagram clearly proving itself wrong

I don't know what to tell you but J with the parentheses upside down still resembles an umbrella. To think that a machine would recognize it's just a flipped umbrella and a human wouldn't is amazing, but here we are. It's doubly baffling because Claude quite clearly explains it in your transcript.

>I feel like I have a pretty good intuition of what's happening here based on my understanding of the underlying mathematical mechanics.

Yes I realize that. I'm telling you that you're wrong.

	▲	striking 11 minutes ago \| parent \| next [-]
		I don't have much more to add to the sibling comment other than the fact that the transcript reads > When you rotate ")" counterclockwise 90°, it becomes a wide, upward-opening arc — like ⌣. but I'm pretty sure that's what you get if you rotate it clockwise.
	▲	pbhjpbhj 29 minutes ago \| parent \| prev [-]
		>Do you think you could publish that in a paper for ext ? You seem to think it's not 'just' tensor arithmetic. Have you read any of the seminal papers on neutral networks, say? It's [complex] pattern matching as the parent said. If you want models to draw composite shapes based on letter forms and typography then you need to train them (or at least fine-tune them) to do that. I still get opposite (antonym) confusion occasionally in responses to inferences where I expect the training data is relatively lacking. That said, you claim the parent is wrong. How would you describe LLM models, or generative "AI" models in the confines of a forum post, that demonstrates their error? Happy for you to make reference to academic papers that can aid understanding your position.

▲

dbdoug 4 hours ago | parent | prev [-]

From Gemini:When you take those two shapes and combine them, the resulting image looks like an umbrella.

▲

hn_acc1 5 hours ago | parent | prev | next [-]

You can't keep pushing the AI hype train if you consider it just a new type of software / fancy statistical database.

▲

menaerus 5 hours ago | parent | prev [-]

Yes, there is - benefit of a doubt.