Do Supervised Distributional Methods Really Learn Lexical Inference Relations?

By Omer Levy, Steffen Remus, Chris Biemann, Ido Dagan. 2015

246 citations recorded by the source corpus.

Distributional representations of words have been recently used in supervised settings for recognizing lexical inference relations between word pairs, such as hypernymy and entailment. We investigate a collection of these state-of-the-art methods, and show that they do not actually learn a relation between two words. Instead, they learn an independent property of a single word in the pair: whether that word is a “prototypical hypernym”.

Subfields: Artificial Intelligence, Cultural Studies, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Representation & Self-Supervised Learning

Citations in the corpus

Annotation Artifacts in Natural Language Inference Data (2018, background): “Levy et al. (2015) demonstrated that supervised lexical inference models rely heavily on artifacts in the datasets, particularly the tendency of some words to serve as prototypical hypernyms.”
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (2019, neutral): “In this paper, we continue recent efforts to understand biases that are introduced during the process of data creation (Levy et al., 2015; Schwartz et al., 2017; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Aharoni and Goldberg, 2018; Paun et al., 2018).”
What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation (2015, background): “In addition, Levy et al. (2015) showed for this task that word embedding-based classiﬁers actually mostly learn whether a word is a general hypernym and not, as would be expected, the relation between two words.”
Detecting Asymmetric Semantic Relations in Context: A Case-Study on Hypernymy Detection (2017, methodology): “We define a 70/5/25 train/dev/test split, and ensure that each set contains different word pairs, to avoid memorization and overfitting (Levy et al., 2015).”
Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS (2018, background): “While this common sense reasoning is trivial for humans, most natural language understanding algorithms do not have the capacity to reason about causal effects not mentioned directly in the surface strings (Levy et al., 2015; Jia & Liang, 2017; Lucy & Gauthier, 2017).”
Large Language Models are not Fair Evaluators (2023, background): “…Liu et al., 2020a,b), question answering (Min et al., 2019), ROC story cloze (Cai, Tu, and Gimpel, 2017; Schwartz et al., 2017), lexical inference (Levy et al., 2015), visual question answering (Goyal et al., 2017), information extraction (Wang et al., 2021, 2022; Song et al., 2023a; Xia et al.,…”
Distributional Models of Word Meaning (2018, background): “Supervised models typically achieve better results, but Levy et al. (2015b) cast doubts on the explanatory adequacy of such approaches.”
A Natural Proof System for Natural Language (2017, background): “Kruszewski and Baroni (2015) tried to model (in)compatible words while the others including Kotlerman et al. (2010); Levy et al. (2015); Shwartz et al. (2016) focused on the hypernymy/entailment relation.”
Named Entity Disambiguation for Noisy Text (2017, methodology): “Finally, we randomly split the data into train (80%), validation (10%), and test (10%), according to website domains in order to minimize lexical memorization (Levy et al., 2015).”
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020, methodology): “Levy et al. (2015) discuss the propensity of classiﬁers to rely on ‘protoypi-cal hypernyms’ in hypernymy detection datasets, and not to solve the detection task.”
Robustness in Coreference Resolution (2020, background): “Luo et al. (2004), Daumé III & Marcu (2005), Yang et al. (2008), Rahman & Ng (2011), Klenner & Tuggener (2011), Ma et al.”