Sparse Overcomplete Word Vector Representations

By Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, Noah A. Smith. 2015

212 citations recorded by the source corpus.

Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “Several studies conducted behavioral experiments to interpret word embeddings by deﬁning intrusion tasks, where humans need to identify an intruder word, chosen based on difference in word embedding dimensions (Murphy et al., 2012; Fyshe et al., 2015; Faruqui et al., 2015).”
Rationalizing Neural Predictions (2016, background): “…analyzing and visualizing state activation (Hermans and Schrauwen, 2013; Karpathy et al., 2015; Li et al., 2016), learning sparse interpretable word vectors (Faruqui et al., 2015b), and linking word vectors to semantic lexicons or word properties (Faruqui et al., 2015a; Herbelot and Vecchi, 2015).”
Visualizing and Understanding Neural Models in NLP (2015, methodology): “Similar strategy is adopted in (Faruqui et al., 2015) by extracting top-ranked words in each vector dimension.”
Understanding Neural Networks through Representation Erasure (2016, background): “…et al., 2015; Murphy et al., 2012), linking dimensions with semantic lexicons or word properties (Faruqui et al., 2014; Herbelot and Vecchi, 2015), learning sparse interpretable word vectors (Faruqui et al., 2015), and empirical studies of LSTM components (Greff et al., 2015; Chung et al., 2014).”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, background): “Faruqui et al. (2015) make word vectors more sparse, and Herbelot and Vec-chi (2015) learn to map distributional word vectors to model-theoretic semantic vectors.”
Feuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships (2016, background): “Other applications for which researchers have prioritized interpretable vector representations include text-to-vision mappings (Lazari-dou et al., 2014) and word embeddings (Fyshe et al., 2015; Faruqui et al., 2015).”
Spatial role labeling with convolutional neural networks (2015, unknown): “Research within the NLP community continues to explore different word embedding models and, in future experiments, it would be interesting to test recent alternatives, as a replacement to the GloVe embeddings used in our experiments [8, 6, 7].”
Contextualized Sparse Representations for Real-Time Open-Domain Question Answering (2019, background): “Related previous work (for a different task) often directly maps dense vectors to a sparse vector space (Faruqui et al., 2015; Subramanian et al., 2018), which can be at most only a few thousand dimensions due to computational cost and small gradients.”
Semantic Structure and Interpretability of Word Embeddings (2017, methodology): “In [26], interpretability of the obtained vector space was evaluated using the word”
Contextualized Sparse Representation with Rectified N-Gram Attention for Open-Domain Question Answering (2019, methodology): “While the sparsiﬁcation of dense word embeddings has been explored previously (Faruqui et al., 2015), its main limitations are that (1) it starts from the dense embedding which might have already lost rich lexical information, and (2) its cardinality is in the order of thousands at max, which we…”
Linear Algebraic Structure of Word Senses, with Applications to Polysemy (2016, methodology): “Sparse coding applied to word embeddings has been used before as a way of getting representations that are more useful in other NLP tasks (Faruqui et al., 2015b), but not in connection with polysemy.”
Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation (2018, background): “For instance, Faruqui et al. (2015b) transform word embeddings into more interpretable sparse representations, obtaining improvements in several benchmark tasks.”
Rotated Word Vector Representations and their Interpretability (2017, methodology): “0 , τ = 10 − 5 , K = 3000 for Glove (Faruqui et al., 2015).”