> There are no "attention heads"

Yes there quite literally is internally in an LLM.

https://bactra.org/notebooks/nn-attention-and-transformers.h...

Just because something is called an "attention head" doesn't mean the terminology makes sense.

If I'm reading the article right, hungryhobbit is accurately describing a single attention head.

And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.

In particular, he is clearly responding to this clause in the comment above:

>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time

Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.