Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I kept that drawing up for many months after, and would gaze at it often during meetings and idle moments in wonder.

This is to say: the autoregressive decoder-only transformer llm architecture as pioneered by openai is wildly simple for how revolutionary its results are. I was reading about non-learned classical SLAM systems (uses video + handcrafted math to produce 3d mappings of physical spaces while also locating the camera in those spaces) at the time, and comparatively speaking I’d say the math is about as complicated as ONE of the components in those complex formulations. The only reason frontier LLMs need 6-figure computers to run is because the model designers made the middle bit in those models REALLY BIG, dimensionally speaking. They just took the steam engine, made a few gargantuan versions of it, and are selling them as the ultimate source of power.

This was openai’s entire breakthrough. Making this particular model architecture larger leads to emergent capabilities like being able to pick the best ending to a story/set of instructions or answer questions about broad factual knowledge. I’ve been meanwhile watching these AI companies attempt, successfully, to sell this capability as some sort of robot consciousness hand-crafted by supergeniuses. The fact that they are getting away with it is almost as shocking to me as the discovery itself.

The part about positional encoding is not correct.

> The intuition: instead of adding position info to each token’s vector, RoPE rotates the vector by an angle that depends on its position

You can't rotate the token's entire vector (or all three vectors, whatever is being implied is unclear). If you change the Value vector of the token based on its position, you break everything. It's specifically avoided. You rotate token 1's Query vector and token 2's Key vector. The rotation applied to both is based on their position, so dot product can be used to compare their relative difference in position.

Either a correction after explaining how the Query, Key and Value vectors work should be in there. Or positional embedding should just be explained after explaining the Query, Key and Value vectors.

If you've only explained how a token's Value vector works at that point in the article, there's no actual foundation taht you could use to explain it.

When the article explains the how the Query and Key vectors work only after that, the reader is building up on a wrong intuition and it gets confusing. Because at no point there's the crucial point about the Query and Key vectors being the ones that change based on the position, not static vectors that the dot product is just applied to as is. RoPE is specifically applied just before the comparison happens, so their positional difference can be accounted for.

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word".

I'm a developer but not very good at maths and I still don't understand any of it.

A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It figured out it could use a radial gradient from the center of the checkbox, and overlay that with a small inner div so you only see the edge that looks like the gradient is circling around the checkbox.

How is that "predicting the next word"?

Not saying AI is intelligent or conscious or anything like that, but the algorithm clearly is far more complex than "predicting words".

What I mean, is the LLM is able to represent things in space . That part I don't understand.

I also still dont understand the relationship between the chat based LLM and the multi modal stuff. I think I read somewhere when image is generated it is also tokens?

This article describes how Transformers work, but not really how LLMs work. Explaining the underlying architecture gives you about as much insight into how a modern LLM behaves as an breakdown of neuronal biochemistry and a few pathways does for the brain. Meaning, almost no insight at all.

I learned TCP/IP by watching and reading raw packets over packet radio at 1200 baud.

I've noticed the same thing is possible if you watch the output of a slow LLM. Eventually you start to see the machinery. input tokens = output tokens, it's math. I can't exactly predict the tokens generated but I can see how they are formed. It's a lot like chess. You can't see every possible move but the mechanism is understandable.

Saying an article is of inferior quality just because editing was AI-assisted is like saying a book is lower quality just because it was printed rather than written by hand

I couldn't load the article directly due to an SSL issue, so here's the archive link:

https://archive.ph/aWtFG

I thought Karpathy’s microgpt explain how LLMs work

this is hard to read...

it goes all over the place.

i'm not actually sure who your target audience is.

there's too many side tangents.

just like, structure it plz.

1. customer feels bad cuz they don't understand how llms work

2. provide high level abstracted explanation (don't dive into concepts yet)

3. provide breakdown guide of overall set of components.

4. walk through each component. don't side track. no need to explain, ROPE,GQA etc... it just distracts.

i.e. customers don't know how llms work, leading them to feel bad about their own intelligence.

at a high level llms take in words, do some math on them, and then produce words, one by one.

inside llms have these different components. we walk through them step by step.

1. tokenizer

2. embedding

3. attention

4. heads

5. ffn

6. sampling

## tokenizer

We are living in a crazy science fiction world where on the top of the HN frontpage there is an article on how LLMs work which is likely itself LLM generated, and the only way to tell is its writing style rather than its factual accuracy.

find it difficult to engage with AI generated text. What am I getting here that I couldn't get from a chatbot.

But how do they “think”? This is the only repo that can tell you that.

https://github.com/space-bacon/SRT

I don't understand how these AI written articles get so many votes.

It sucks that this article is clearly LLM edited, with common phrases like "same shape as", "the intuition: ", and the "tiny explainer" which clearly generalized from a prompt accidentally.

Good article, but when sharing it I will have to preface "yes it's slop, but it's a good explanation".

Absolutely embarrassing that the author didn't catch that these LLM-isms are a (and here I'll use one) bad signal.

In fact, I would go so far as to say that publishing in this style stems from a lack of reading experience and writing experience, which does not bode well for someone pretending to be an expert. I gave this article to someone highly intelligent who doesn't know the first thing about how LLMs work internally, and she immediately called out that it reads like AI text.

What's the point of this? Im not here to engage with AI bots

accidentally quadratic

Next do "why LLMs work"