Do Neural Language Representations Learn Physical Commonsense?

By Maxwell Forbes, Ari Holtzman, Yejin Choi. 2019

115 citations recorded by the source corpus.

Humans understand language based on the rich background knowledge about how the physical world works, which in turn allows us to reason about the physical world through language. In addition to the properties of objects (e.g., boats require fuel) and their affordances, i.e., the actions that are applicable to them (e.g., boats can be driven), we can also reason about if-then inferences between what properties of objects imply the kind of actions that are applicable to them (e.g., that if we can drive something then it likely requires fuel). In this paper, we investigate the extent to which state-of-the-art neural language representations, trained on a vast amount of natural language text, demonstrate physical commonsense reasoning. While recent advancements of neural language models have demonstrated strong performance on various types of natural language inference tasks, our study based on a dataset of over 200k newly collected annotations suggests that neural language representations still only learn associations that are explicitly written down.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Measuring and Improving Consistency in Pretrained Language Models (2021, background): “, 2020), commonsense (Forbes et al., 2019; Da and Kasai, 2019; Zhang et al., 2020) and reasoning (Talmor et al.”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, neutral): “, 2019), and commonsense (Forbes et al., 2019; Zhou et al., 2019; Weir et al., 2020).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Forbes et al. (2019) show that BERT can “guess” the affordances and properties of many objects, but does not have the information about their interactions (e.g. it “knows” that people can walk into houses, and that houses are big, but it cannot infer that houses are bigger than people.)”
Pre-Trained Models: Past, Present and Future (2021, background): “In addition to the commonsense features/attributes, the implicit relations between different attributes are important and Forbes et al. (2019) show that current PTMs ’ representations cannot model the implicit relations well, which requires further exploration.”
Inducing Relational Knowledge from BERT (2019, neutral): “More closely related to our work, Forbes, Holtzman, and Choi (2019) analyze to what extent properties and affordances of objects can be predicted using neural language models, by relying on manually chosen sentences.”
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020, background): “There have been several such studies: Forbes et al. (2019) study physical commonsense encoded in BERT.”
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (2019, background): “Forbes et al. (2019) investigates the commonsense aptitude of contextual representations.”
Do Neural Language Models Overcome Reporting Bias? (2020, background): “Both performed better on ency-clopedic and functional properties (“A bear is an animal”) than on perceptual properties, which are less often mentioned in text (Collell Talleda and Moens, 2016; Forbes et al., 2019).”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “Several recent papers propose to use natural language for explanation and commonsense reasoning (Lei et al., 2016; Camburu et al., 2018; Forbes and Choi, 2017; Chai et al., 2018; Forbes et al., 2019; Rajani et al., 2019; DeYoung et al., 2020).”
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color (2021, background): “This includes, for instance, some ∗ For correspondence: {abdou,soegaard}@di.ku.dk features of concrete and abstract concepts, such as objects’ attributes and affordances (Forbes et al., 2019b; Weir et al., 2020).”
Probing Contextual Language Models for Common Ground with Visual Representations (2020, background): “From factual (Petroni et al., 2019; Jawahar et al., 2019; Roberts et al., 2020) to linguistic (Conneau et al., 2018; Liu et al., 2019a; Talmor et al., 2019) and commonsense (Forbes et al., 2019) knowledge, a wide set of properties have been previously analysed.”
Probing Text Models for Common Ground with Visual Representations (2020, background): “From factual (Petroni et al., 2019; Jawahar et al., 2019; Roberts et al., 2020) to linguistic (Conneau et al., 2018b; Liu et al., 2019a; Talmor et al., 2019) and common-sense (Forbes et al., 2019) knowledge, a wide set of properties have been previously analysed.”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, unknown): “However, several studies [Forbes et al., 2019, Zhou et al., 2020b, Lin et al., 2020, Boratko et al., 2020, Singh et al., 2021] show that when the datasets are speciﬁcally changed to target the weaknesses of transformer-based models (for example, by adversarial instances), the models fail.”