Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks

By Ting-Yun Chang, Yang Liu, Karthik Gopalakrishnan, Behnam Hedayatnia, Pei Zhou, Dilek Z. Hakkani-Tür. 2020

41 citations recorded by the source corpus.

Pretrained language models have excelled at many NLP tasks recently; however, their social intelligence is still unsatisfactory. To enable this, machines need to have a more general understanding of our complicated world and develop the ability to perform commonsense reasoning besides fitting the specific downstream tasks. External commonsense knowledge graphs (KGs), such as ConceptNet, provide rich information about words and their relationships. Thus, towards general commonsense learning, we propose two approaches to implicitly and explicitly infuse such KGs into pretrained language models. We demonstrate our proposed methods perform well on SocialIQA, a social commonsense reasoning task, in both limited and full training data regimes.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “These existing solutions tackle content-related harms, but future work in reducing discrimination and toxicity in LMs may also benefit from KBs that encode social (Chang et al., 2020), cultural, (Hershcovich et al., 2022), and moral norms (Hendrycks et al., 2021; Jiang et al., 2021).”
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021, background): “Injecting commonsense knowledge into NLP tasks is gaining attention (Storks et al., 2019; Chang et al., 2020).”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “Yet these methods either retrieve from in-domain knowledge bases (Mitra et al., 2019; Chang et al., 2020) that do not have good coverage over commonsense, or generate knowledge from neural models (Shwartz et al., 2020; Gu et al., 2022; Liu et al., 2022), which often need domain-speciﬁc engineering…”
Kformer: Knowledge Injection in Transformer Feed-Forward Layers (2022, methodology): “At the representation level, the model obtains the knowledge representation and the text representation respectively and interacts with these representations using attention methods [Chang et al., 2020].”
Explaining Toxic Text via Knowledge Enhanced Text Generation (2022, methodology): “Following Chang et al. (2020), given a BART model and an input, we extract ranked triples re-lated to the input post and keep the top k triples per post where we vary the k ∈ { 3 , 5 , 10 , 15 , 20 , 25 } .”
Generated Knowledge Prompting for Commonsense Reasoning (2021, methodology): “Consequently, these models can directly perform inference on tasks like commonsense reasoning (Trinh and Le, 2018; Yang et al., 2020), text classiﬁcation (Shin et al., 2020; Puri and Catanzaro, 2019), and natural language inference (Shin et al., 2020; Schick and Schütze, 2021).”