| ▲ | ricksunny a month ago | |
Vaswani, A., et al. (2017) Attention Is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, 4-9 December 2017, 6000-6010. Generally understood to be an output of Googlers. | ||
| ▲ | biophysboy a month ago | parent [-] | |
Transformers are an applied science: https://patents.google.com/patent/US10740433B2/en Basic research would be something like optimal control theory, which came well before the transformer design. I'm not trying to be evasive; I can see how my distinction could be seen as conveniently just outside industry's purview. Put it this way: I think companies, particularly small ones, are incentivized to pursue well-known methods/materials. Innovation modulates and optimizes. | ||