Language Generation with Multi-hop Reasoning on Commonsense Knowledge Graph

By Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, Minlie Huang. 2020

120 citations recorded by the source corpus.

Despite the success of generative pre-trained language models on a series of text generation tasks, they still suffer in cases where reasoning over underlying commonsense knowledge is required during generation. Existing approaches that integrate commonsense knowledge into generative pre-trained language models simply transfer relational knowledge by post-training on individual knowledge triples while ignoring rich connections within the knowledge graph. We argue that exploiting both the structural and semantic information of the knowledge graph facilitates commonsense-aware text generation. In this paper, we propose Generation with Multi-Hop Reasoning Flow (GRF) that enables pre-trained models with dynamic multi-hop reasoning on multi-relational paths extracted from the external commonsense knowledge graph. We empirically show that our model outperforms existing baselines on three text generation tasks that require reasoning over commonsense knowledge. We also demonstrate the effectiveness of the dynamic multi-hop reasoning module with reasoning paths inferred by the model that provide rationale to the generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Wang et al. (2021a); Ji et al. (2020) also use KBs to embed commonsense knowledge in LMs.”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “These models are most effective for tasks that require understanding the entire sentence, such as text classification [26] and named entity recognition [53].”
Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “• Knowledge-augmented NLU: (Zhang et al., 2019; Peters et al., 2019; Liu et al., 2020; Ding et al., 2019; Lv et al., 2020; Yu et al., 2022b); • Knowledge-augmented NLG: (Zhou et al., 2018; Zhang et al., 2020a; Ji et al., 2020b; Lewis et al., 2020; Wang et al., 2021); • Commonsense Knowledge and Reas”
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022, methodology): “Ji et al. (2020) performed dynamic multi-hop reasoning on multi-relational paths extracted from the external commonsense KG. Recently, some work attempted to integrate exter-nal commonsense knowledge into generative pre-trained language models (Guan et al., 2020; Bhaga-vatula et al., 2020; Liu et…”
Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks (2020, background): “Recently there is an increasing effort to utilize external knowledge bases to incorporate commonsense information underlying the text (Shwartz et al., 2020; Mitra et al., 2019; Ji et al., 2020a,b).”
OTTers: One-turn Topic Transitions for Open-Domain Dialogue (2021, unknown): “…linking to a KG. • We show that our KG-grounded dataset can effectively leverage the reasoning component of an existing Transformer-based model (Ji et al., 2020) to generate better output compared to a vanilla GPT-2 (Radford et al., 2019) decoder, both in in-domain and out-of-domain data…”
Behavior Cloned Transformers are Neurosymbolic Reasoners (2022, background): “While large language models are capable of a variety of common sense reasoning abilities (Liu et al., 2022b; Ji et al., 2020), contemporary agents typically struggle on tasks such as navigation, arithmetic, knowledge base lookup, and other tasks that humans typically make use of external tools (such”
A Survey of Knowledge-enhanced Text Generation (2020, background): “, answer) Creative writing Document - - [38] - [56, 70] - - (e.”
A Review on Language Models as Knowledge Bases (2022, methodology): “While not explicitly a model, GRF (Ji et al., 2020) propose a method to enable multi-hop reasoning with transformers.”