Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations

By Jeff Da, Jungo Kasai. 2019

41 citations recorded by the source corpus.

Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models. Thus, we investigate and challenge several aspects of BERT’s commonsense representation abilities. First, we probe BERT’s ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas. Next, we show that, by augmenting BERT’s pretraining data with additional data related to the deficient attributes, we are able to improve performance on a downstream commonsense reasoning task while using a minimal amount of data. Finally, we develop a method of fine-tuning knowledge graphs embeddings alongside BERT and show the continued importance of explicit knowledge graphs.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Mohit Bansal
3. Hinrich Schütze
4. Minlie Huang
5. Xipeng Qiu
6. Yoav Goldberg
7. Yonatan Belinkov
8. Maosong Sun

In Large Language Models

1. Hinrich Schütze
2. Zhiyuan Liu
3. Jie Tang
4. Yoav Goldberg
5. Danai Koutra
6. Tara Safavi
7. Xiao Liu
8. Minlie Huang

In Prompting & Reasoning

1. Minlie Huang
2. Zhiyuan Liu
3. Jie Tang
4. Ning Ding
5. Xiao Liu
6. Jiezhong Qiu
7. Xu Han
8. Zhengyan Zhang

Citations in the corpus

• Measuring and Improving Consistency in Pretrained Language Models (2021, background): “This definition is in the spirit of the RTE definition (Dagan et al., 2005), which similarly supports a more flexible use of the notion of entailment.”
• Pre-Trained Models: Past, Present and Future (2021, background): “Similarly, Da and Kasai (2019) also find that PTMs have learned various commonsense features in their representation space based on a series of probing tasks.”
• On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020, background): “Da and Kasai (2019) probe BERT for its understanding of object attributes, finding that it learns physical concrete norms ( is made of wood ) better than abstract ones ( is strong ).”
• Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “modeling corpora like Wikipedia are known to contain KB-like assertions about the world (Da and Kasai, 2019).”