Word Embeddings as Metric Recovery in Semantic Spaces

By Tatsunori B. Hashimoto, David Alvarez-Melis, Tommi S. Jaakkola. 2016

90 citations recorded by the source corpus.

Continuous word representations have been remarkably useful across NLP tasks but remain poorly understood. We ground word embeddings in semantic spaces studied in the cognitive-psychometric literature, taking these spaces as the primary objects to recover. To this end, we relate log co-occurrences of words in large corpora to semantic similarity assessments and show that co-occurrences are indeed consistent with an Euclidean semantic space hypothesis. Framing word embedding as metric recovery of a semantic space unifies existing word embedding algorithms, ties them to manifold learning, and demonstrates that existing algorithms are consistent metric recovery methods given co-occurrence counts from random walks. Furthermore, we propose a simple, principled, direct metric recovery algorithm that performs on par with the state-of-the-art word embedding and manifold learning methods. Finally, we complement recent focus on analogies by constructing two new inductive reasoning datasets—series completion and classification—and demonstrate that word embeddings can be used to solve them as well.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Generating Sentences by Editing Prototypes (2017, background): “More generally, results in manifold learning demonstrate that a weak metric such as lexical similarity can be used to extract semantic similarity through distributional statistics (Tenenbaum et al., 2000; Hashimoto et al., 2016).”
Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec (2017, methodology): “torization, which provides us with a tool to analyze the above network embedding models; Arora et al . [1] propose a generative model RAND-WALK to explain word embedding models; and Hashimoto et al . [18] frame word embedding as a metric learning problem. Built upon the work in [ 24 ], we theoretica”
Probabilistic Bias Mitigation in Word Embeddings (2019, background): “[9], although our framework can be equivalently instantiated for most other popular embedding methods, owing to their core similarities [7, 10].”
Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning (2020, background): “[31] assume a generative language model where words are emitted by a random walk.”
Gromov-Wasserstein Alignment of Word Embedding Spaces (2018, background): “Indeed, word embeddings are estimated primarily in a relational manner to the extent that the algorithms are naturally interpreted as metric recovery methods (Hashimoto et al., 2016).”
A Plug-and-Play Method for Controlled Text Generation (2021, methodology): “This method not only encourages the explicit appearance of a guide word, but also encourages the model to generate appropriate context for the guide word to appear, i.e., words in the same semantic neighborhood (Hashimoto et al., 2016).”
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2021, methodology): “…tasks while pro-1 Nearest-neighbor projection via dot product has been previously used to study properties of continuous word embeddings (Mikolov et al., 2013; Hashimoto et al., 2016) and is commonly performed in the ﬁnal layer of modern generative LMs (Radford et al., 2019; Raffel et al., 2020).”