Linear Algebraic Structure of Word Senses, with Applications to Polysemy

By Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski. 2016

346 citations recorded by the source corpus.

Word embeddings are ubiquitous in NLP and information retrieval, but it is unclear what they represent when the word is polysemous. Here it is shown that multiple word senses reside in linear superposition within the word embedding and simple sparse coding can recover vectors that approximately capture the senses. The success of our approach, which applies to several embedding methods, is mathematically explained using a variant of the random walk on discourses model (Arora et al., 2016). A novel aspect of our technique is that each extracted word sense is accompanied by one of about 2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense. Discourse atoms can be of independent interest, and make the method potentially more useful. Empirical tests are used to verify and support the theory.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Gender-Neutral Word Embeddings (2018, background): “…have been made by leveraging semantic lexicons and morphology (Luong et al., 2013; Faruqui et al., 2014), disambiguating multiple senses (ˇSuster et al., 2016; Arora et al., 2018; Upadhyay et al., 2017), and modeling contextualized information by deep neural networks (Peters et al., 2018).”
Bias in word embeddings (2020, background): “Arora et al. [4] prove that different meanings of words are ‘encoded’ in word embeddings and can be retrieved.”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “As first observed by (Arora et al., 2018), superposition involves compressing multiple features into a smaller number of dimensions (Elhage et al., 2022b).”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Such framework has allowed researchers to better understand grokking (Nanda et al., 2023), superposition (Elhage et al., 2022b; Scherlis et al., 2022; Arora et al., 2018), or even individual neurons (Mu and Andreas, 2020; Antverg Belinkov, 2021; Gurnee et al., 2023).”
Making Method of Moments Great Again? -- How can GANs learn the target distribution (2020, methodology): “ur study to other popular target distributions such as 15 (a) Wasserstein Loss (b) Parameter Distance Figure 7: Discriminators as 2-layer ReLU Networks sparse coding/non-negative matrix factorization [18, 19, 49, 47, 7] 16 9 Omitted Proofs in Section 6 9.1 Some Simple Lemmas and Properties Before we”
Semantic Structure and Interpretability of Word Embeddings (2017, methodology): “Instead of learning interpretable word representations directly from co-occurrence matrices, [25] and [26] proposed to use sparse coding techniques on conventional dense word embeddings to obtain sparse, higher dimensional and more interpretable vector spaces.”
A Latent Variable Model Approach to PMI-based Word Embeddings (2015, background): “For example, it led to an understanding of how the different senses of a polysemous word (e.g., bank ) reside in linear superposition within the word embedding (Arora et al., 2016).”
SenseBERT: Driving Some Sense into BERT (2019, background): “Such embeddings were shown to contain some amount of word-sense information (Iacobacci et al., 2016; Yuan et al., 2016; Arora et al., 2018; Le et al., 2018).”
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks (2017, background): “By encoding information about words in a relatively simple algebraic structure [Arora et al., 2016] they enable fast transfer to the task of interest 1 .”
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, background): “…many mathematical properties of embeddings: embeddings are equivalent to count-based matrices (Levy and Goldberg, 2014b), expected to be linearly dependant (Arora et al., 2016), expressible as a unitary matrix (Smith et al., 2017) or as a perturbation of an identity matrix (Yin and Shen, 2018).”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, methodology): “Ex- tensions of the mathematical formulation, such as the use of word weighting when building context vectors as in Arora et al. (2018b) or of spectral information along the lines of Mu and Viswanath (2018), are also worthy of further study.”