Remix.run Logo
▲ microtonal 4 hours ago

Not to downplay his accomplishments (I have sung my praises in another comment), but even vector models were already around and somewhat popular before. Much of it started in Information Retrieval with things like TF-IDF vectors and later PPMI matrices that were converted to small, dense matrices with SVD. Of course, later Yoshua Bengio et al. introduced neural language models for training embeddings.

There is even a paper by Levy & Goldberg from 2014 that shows that skip-gram with negative sampling implicitly factorizes a PPMI word-context matrix.

But Mikolov's formulation is very nice in that it is a simple linear model that allows for a very simple Hogwild optimization that worked very well/fast on multi-core CPUs of the time.

▲DC-3 4 hours ago | parent [-]

Just FYI: You sang his praises. If you "sang my praises", that means you celebrated yourself.