Are Distributional Representations Ready for the Real World? Evaluating Word Vectors for Grounded Perceptual Meaning

By Li Lucy, Jon Gauthier. 2017

65 citations recorded by the source corpus.

Distributional word representation methods exploit word co-occurrences to build compact vector encodings of words. While these representations enjoy widespread use in modern natural language processing, it is unclear whether they accurately encode all necessary facets of conceptual meaning. In this paper, we evaluate how well these representations can predict perceptual and conceptual features of concrete concepts, drawing on two semantic norm datasets sourced from human participants. We find that several standard word representations fail to encode many salient perceptual features of concepts, and show that these deficits correlate with word-word similarity prediction errors. Our analyses provide motivation for grounded and embodied language learning approaches, which may help to remedy these deficits.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “Physical knowledge Prior work has studied learning grounded knowledge about objects and verbs: from knowledge bases (Li et al., 2016), syntax parses (Forbes and Choi, 2017), word embed-dings (Lucy and Gauthier, 2017), and images and dictionary deﬁnitions (Zellers and Choi, 2017).”
Experience Grounds Language (2020, background): “Past work has found that pretrained word and sentence representations fail to capture many grounded features of words (Lucy and Gauthier, 2017) and sentences, and current NLU sys-”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Also known as reporting bias (Gordon and Van Durme, 2013; Lucy and Gauthier, 2017), this phenomenon likely limits LLMs’ ability to learn social commonsense knowledge from static text.”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “For example, [39] has recently tried to evaluate how well the word vectors capture the necessary facets of conceptual meaning.”
Do Neural Language Representations Learn Physical Commonsense? (2019, positive): “We take the set of objects and properties from McRae et al. (2005) and perform ﬁltering and preprocessing similar to Lucy and Gauthier (2017).”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “For example, Lucy and Gauthier [39] has recently tried to evaluate how well the word vectors capture the necessary facets of conceptual meaning.”
Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS (2018, background): “While this common sense reasoning is trivial for humans, most natural language understanding algorithms do not have the capacity to reason about causal effects not mentioned directly in the surface strings (Levy et al., 2015; Jia & Liang, 2017; Lucy & Gauthier, 2017).”
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (2019, methodology): “For baseline tests, we use the pretrained 300 dimensional GloVe embeddings,1 as they have shown to perform better than word2vec embeddings (Lucy and Gauthier, 2017).”
Probing Neural Language Models for Human Tacit Assumptions (2020, background): “However, the empirical ﬁnding coheres with previous conclusions that models that learn from language alone lack knowledge of perceptual features (Collell & Moens, 2016; Lucy & Gauthier, 2017).”
Probing Text Models for Common Ground with Visual Representations (2020, background): “Closer to our motivations of understanding what is encoded in already trained text models are the works of (Lucy and Gauthier, 2017) and (Scialom et al.”