Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.

I get that this is for explaining GPT-2, but I really hope laymen don't use it as an example of how modern models work (ex. absolute positional encoding is no longer used)

edit: I know that it mentions its not modern, but these kinds of details have major implications in terms of the representations a model can learn, which is in many ways the most important part!

I never understood the thinking behind the separate key query value matrixes? What are they doing exactly?

Great site, intuitive description. I also found [1] very useful in the past.

[1] https://bbycroft.net/llm

Damn that page took down my Chromebook, never happened before..

Nice work. I really appreciate this tool for enhancing my limited understanding the mechanism(s) behind attention and LLMs.

This is not at all what I was hoping for. Expected a lot more Unicron.