Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

By W. Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, Meng Jiang. 2022

39 citations recorded by the source corpus.

Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.

Subfields: Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Xiang Ren
2. Jiawei Han
3. Bill Yuchen Lin
4. Chenguang Zhu
5. Heng Ji
6. Denny Zhou
7. Jason Wei
8. Meng Jiang

In Prompting & Reasoning

1. Denny Zhou
2. Xuezhi Wang
3. Jason Wei
4. Ed H. Chi
5. Quoc V. Le
6. Wenhu Chen
7. Dale Schuurmans
8. Xiang Ren

Citations in the corpus

• Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “…importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths (Chen et al., 2019), or by the introduction of latent variables in a commonsense knowledge graph (Yu et al., 2022).”
• Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “Yu et al. (2022c) and Xu et al. (2021) ∗ propose to use dictionary descriptions as additional knowledge source for natural language understanding and commonsense reasoning tasks.”
• Large Language Models on Graphs: A Comprehensive Survey (2023, background): “In recent years, large language models (LLMs) have demonstrated impressive capabilities in tasks such as machine translation, text summarization, and question answering [26], [43], [112]–[115], [195].”