Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.

By Anna Rogers, Aleksandr Drozd, Satoshi Matsuoka. 2016

275 citations recorded by the source corpus.

Following up on numerous reports of analogy-based identiﬁcation of “linguistic regularities” in word embeddings, this study applies the widely used vector offset method to 4 types of linguistic relations: inﬂectional and derivational morphology, and lexicographic and en-cyclopedic semantics. We present a balanced test set with 99,200 questions in 40 categories, and we systematically examine how accuracy for different categories is affected by window size and dimensionality of the SVD-based word embeddings. We also show that GloVe and SVD yield similar patterns of results for different categories, offering further evidence for conceptual similarity between count-based and neural-net based models.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, methodology): “Moreover, the extended analogy test set created by Gladkova, Drozd, and Matsuoka (2016) also includes examples where B is the correct answer, for example country:language and thing:color .”
Inducing Relational Knowledge from BERT (2019, background): “…of instances per relation (Mikolov et al. 2013); • the Bigger Analogy Test Set (BATS), which contains 40 relations with 50 instances per relation (Gladkova, Drozd, and Matsuoka 2016); • the DiffVec Test Set (DV) contains 36 relations with a varying number of instances per relation (Vylomova…”
Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen (2016, background): “However, accuracy of solving word analogies also varies immensely for different linguistic relations (Gladkova et al., 2016).”
The (too Many) Problems of Analogical Reasoning with Word Vectors (2017, background): “Given the above, the question is: if analogical reasoning requires much case-by-case consideration in humans, what should we expect from VSMs with a single linear algebra operation?”
On learning and representing social meaning in NLP: a sociolinguistic perspective (2021, background): “These representations are shown to capture a range of linguistic phenomena (Baroni et al., 2014; Conneau et al., 2018; Gladkova et al., 2016).”
deb2viz: Debiasing gender in word embedding data using subspace visualization (2020, methodology): “Also, the plot clearly shows that the word vectors of the GP-[15] and the GN-[11] word2vec , still exhibit gender stereotype.”
BYU-EVE: Mixed Initiative Dialog via Structured Knowledge Graph Traversal and Conversational Scaffolding (2018, background): “have shown and other researchers have confirmed that within these embedding spaces, simple mathematical operations are sufficient to solve analogical queries [20, 27, 11, 12].”
Issues in evaluating semantic spaces using word analogies (2016, result): “Our results also highlight the difﬁculty in comparing spaces based on accuracy measures averaged across heterogeneous and unbalanced analogy sets (Gladkova et al., 2016).”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, methodology): “In Figure 12, we include an analog of Figure 5 where we add the embedding representation α ¯ λ C (4.1) for each of the 27 concepts to λ ( x j ) and see the change in logits.”
Analogical Reasoning on Chinese Morphological and Semantic Relations (2018, background): “To avoid the imbalance problem addressed in English benchmarks (Gladkova et al., 2016), we set a limit of 50 word pairs at most for each relation.”
Harvesting Common-sense Navigational Knowledge for Robotics from Uncurated Text Corpora (2017, background): “This tendency of analogy operations to produce results close to their origins is well documented and easily verifiable [21, 17].”
Distributional Effects of Gender Contrasts Across Categories (2019, background): “For instance, Gladkova et al. (2016) have highlighted that “derivational and lexicographic relations remain a major challenge”; Levy and Goldberg (2014) and Linzen (2016) both stress that simple additive models do not suffice to model 3The utility of vector shifts as inputs to word relation…”
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (2018, methodology): “We use the Bigger Analogy Test Set (BATS) (Gladkova et al., 2016) which contains four groups of relations: encyclopedic semantics (e.”
Metrology for AI: From Benchmarks to Instruments (2019, background): “…is collected(Son 2018; Inel 2017; 2019) • results from the evaluations of AI systems against these benchmarks are presented(Rogers, Drozd, and Li 2017; Gladkova, Drozd, and Matsuoka 2016; Wendlandt, Kummerfeld, and Mihalcea 2018; Ramakrishnan et al. 2018) According to (Baker 2016) we experience a…”
Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types (2020, methodology): “The ability of each model to capture semantic similarity, relatedness as well as morphological, lexical, encyclopedic and lexicographic analogies [11] was measured (see Table 1).”