Parallelizing Word2Vec in Shared and Distributed Memory

By Shihao Ji, Nadathur Rajagopalan Satish, Sheng R. Li, Pradeep K. Dubey. 2016

72 citations recorded by the source corpus.

Word2vec is a widely used algorithm for extracting low-dimensional vector representations of words. State-of-the-art algorithms including those by Mikolov et al. [1] , [2] have been parallelized for multi-core CPU architectures, but are based on vector-vector operations with “Hogwild” updates that are memory-bandwidth intensive and do not efficiently use computational resources. In this paper, we propose “HogBatch” by improving reuse of various data structures in the algorithm through the use of minibatching and negative sample sharing, hence allowing us to express the problem using matrix multiply operations. We also explore different techniques to distribute word2vec computation across nodes in a computer cluster, and demonstrate good strong scalability up to 32 nodes. The new algorithm is particularly suitable for modern multi-core/many-core architectures, especially Intel's latest Knights Landing processors, and allows us to scale up the computation near linearly across cores and nodes, and process hundreds of millions of words per second, which is the fastest word2vec implementation to the best of our knowledge. We released the source code for reproducible research and general usage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

On Sampling Strategies for Neural Network-based Collaborative Filtering (2017, background): “We also find that the idea of sharing negative examples is exploited to speed up word embedding training in [14].”
The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings (2018, background): “Word2vec can achieve success rates as high as 74% (Ji et al. 2016) on a challenging analogy test comprising 20 thousand questions involving semantic comparisons ranging from currency-country (kwanza is to Angola as rial is to Iran) and male-female (man is to woman as waiter is to waitress) t”
LASAGNE: Locality and Structure Aware Graph Node Embedding (2017, unknown): “For details see [35].”