| ▲ | Alifatisk a day ago | ||||||||||||||||
> Attention was developed before transformers. I just looked this up and it’s true, this changes the timeline I had in my mind completely! I thought the paper on Transformers is what also introduced the attention mechanism, but it existed before too and was applied on RNN encoder-decoder. Wow | |||||||||||||||||
| ▲ | logicchains a day ago | parent [-] | ||||||||||||||||
Knowing how such things go, it was probably invented by Schmidhuber in the 90s. | |||||||||||||||||
| |||||||||||||||||