Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations
By Alexandre Salle, Aline Villavicencio, Marco Idiart. 2016
82 citations recorded by the source corpus.
In this paper, we propose LexVec, a new method for generating distributed word representations that uses low-rank, weighted factorization of the Positive Point-wise Mutual Information matrix via stochastic gradient descent, employing a weighting scheme that assigns heavier penalties for errors on frequent co-occurrences while still accounting for negative co-occurrence. Evaluation on word similarity and analogy tasks shows that LexVec matches and often outperforms state-of-the-art methods on many of these tasks.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Timo Schick
- Hinrich Schütze
- Chenguang Zhu
- Weizhu Chen
- Mohit Bansal
- Marco Idiart
- Aline Villavicencio
- Alexandre Salle
In Natural Language Processing
- Timo Schick
- Hinrich Schütze
- Chenguang Zhu
- Weizhu Chen
- Marco Idiart
- Aline Villavicencio
- Alexandre Salle
- Michal Konkol
In Representation & Self-Supervised Learning
- Timo Schick
- Hinrich Schütze
- Marco Idiart
- Alexandre Salle
- Aline Villavicencio
- Michal Konkol
- Tomas Brychcin
- Weizhu Chen
Citations in the corpus
- Geographical Evaluation of Word Embeddings (2017, methodology): “LexVec is based on factorization of positive point-wise mutual information matrix using proven strategies from GloVe, SkipGram, and methods based on singular value decomposition (Salle et al., 2016).”
- Learning Semantic Representations for Novel Words: Leveraging Both Form and Context (2018, methodology): “In a similar fashion, Salle and Villavicencio (2018) incorporate n -grams and morphemes into the LexVec model (Salle, Idiart, and Villavicencio 2016).”
- Enhancing the LexVec Distributed Word Representation Model Using Positional Contexts and External Memory (2016, methodology): “In this paper, we focus on improving a state-ofthe-art counting model, LexVec (Salle et al., 2016), which performs factorization of the positive pointwise mutual information(PPMI) matrix usingwindow sampling and negative sampling (WSNS).”
- Zero-training Sentence Embedding via Orthogonal Basis (2018, background): “The concept of word embeddings has been prevalent in NLP community in recent years, as they can characterize semantic similarity between any pair of words, achieving promising results in a large number of NLP tasks (Mikolov et al., 2013; Pennington et al., 2014; Salle et al., 2016).”