Swivel: Improving Embeddings by Noticing What's Missing

By Noam Shazeer, Ryan Doherty, Colin Evans, Chris Waterson. 2016

81 citations recorded by the source corpus.

We present Submatrix-wise Vector Embedding Learner (Swivel), a method for generating low-dimensional feature embeddings from a feature co-occurrence matrix. Swivel performs approximate factorization of the point-wise mutual information matrix via stochastic gradient descent. It uses a piecewise loss with special handling for unobserved co-occurrences, and thus makes use of all the information in the matrix. While this requires computation proportional to the size of the entire matrix, we make use of vectorized multiplication to process thousands of rows and columns at once to compute millions of predicted values. Furthermore, we partition the matrix into shards in order to parallelize the computation across many nodes. This approach results in more accurate embeddings than can be achieved with methods that consider only observed co-occurrences, and can scale to much larger corpora than can be handled with sampling methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah Constant
2. Matthew Henderson
3. Sanjiv Kumar
4. Brian Strope
5. R. Kurzweil
6. Mandy Guo
7. Yun-Hsuan Sung
8. Rami Al-Rfou

In Computer Vision and Pattern Recognition

1. Sanjiv Kumar
2. Ruiqi Guo
3. Rami Al-Rfou
4. Matthew Henderson
5. Balint Miklos
6. László Lukács
7. R. Kurzweil
8. Yun-Hsuan Sung

In Representation & Self-Supervised Learning

1. Rami Al-Rfou
2. Matthew Henderson
3. Sanjiv Kumar
4. Heeyoung Lee
5. R. Kurzweil
6. Yun-Hsuan Sung
7. Brian Strope
8. Maria Antoniak

Citations in the corpus

• Efficient Natural Language Response Suggestion for Smart Reply (2017, background): “Vector representations of words, or word embeddings , have been widely adopted, particularly since the introduction of efficient computational learning algorithms that can derive meaningful embeddings from unlabeled text [15, 17, 20].”
• Evaluating the Stability of Embedding-based Word Similarities (2018, methodology): “The training corpus is merely a sample of the authors’ language model (Shazeer et al., 2016).”
• Bridging the Gap for Tokenizer-Free Language Models (2019, methodology): “We report the best value achieved across all layers versus Swivel (Shazeer et al., 2016).”