Generated Knowledge Prompting for Commonsense Reasoning

By Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi. 2021

407 citations recorded by the source corpus.

It remains an open question whether incorporating external knowledge benefits commonsense reasoning while maintaining the flexibility of pretrained sequence models. To investigate this question, we develop generated knowledge prompting, which consists of generating knowledge from a language model, then providing the knowledge as additional input when answering a question. Our method does not require task-specific supervision for knowledge integration, or access to a structured knowledge base, yet it improves performance of large-scale, state-of-the-art models on four commonsense reasoning tasks, achieving state-of-the-art results on numerical commonsense (NumerSense), general commonsense (CommonsenseQA 2.0), and scientific commonsense (QASC) benchmarks. Generated knowledge prompting highlights large-scale language models as flexible sources of external knowledge for improving commonsense reasoning.Our code is available at github.com/liujch1998/GKP

Subfields: Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Rationale-Augmented Ensembles in Language Models (2022, background): “For commonsense reasoning tasks, Liu et al. (2022a) generate relevant knowledge as additional inputs to the model, to improve the performance.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “Shwartz et al. (2020); Betz et al. (2021); Liu et al. (2022) use a pretrained causal LM to generate context to improve performance on multiple choice commonsense tasks.”
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks (2022, background): “Similar limitations were observed in language related compounded tasks, including commonsense reasoning (Liu et al., 2022; Wei et al., 2022; Zelikman et al., 2022), math word problems (Pi˛ekos et al., 2021; Wei et al., 2022), and programs execution (Nye et al., 2022).”
Reasoning with Language Model Prompting: A Survey (2022, background): “, 2022b), generated knowledge prompting (Liu et al., 2022c), which can dramatically narrow the gap between human and machine intelligence.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “The “generated knowledge” [61] approach in prompt engineering is a technique that leverages the ability of LLMs to generate potentially useful information about a given question or prompt before generating a final response.”
Rethinking with Retrieval: Faithful Large Language Model Inference (2022, methodology): “Liu et al. (2022) further propose generating knowledge from LLMs, which is then used as additional input to improve commonsense reasoning.”
What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, background): “[30] [18] provides segmentation maps of ImageNet images broken down into parts.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “Prompting has been successful for a variety of natural language tasks [37], including but not limited to classification tasks [42, 56], semantic parsing [44, 74], knowledge generation [35, 57], and dialogue systems [20, 28].”
Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “A line of recent work has shown that relevant knowledge can be elicited from large language models, especially for those domains that lack appropriate knowledge bases with sufficient coverage (Liu et al., 2022; Fang et al., 2022).”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, methodology): “…retrieve true E s (the set of generated E s that are inferred to be True by M AIEUTIC PROMPTING ), then evaluate them over the four criteria from Liu et al. (2021): (1) Grammaticality of the explanations, (2) Relevance of the explanations to the question, (3) Factuality : whether the…”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “…bases (Mitra et al., 2019; Chang et al., 2020) that do not have good coverage over commonsense, or generate knowledge from neural models (Shwartz et al., 2020; Gu et al., 2022; Liu et al., 2022), which often need domain-speciﬁc engineering and very large models (e.g. GPT-3 (Brown et al., 2020)).”
Social Simulacra: Creating Populated Prototypes for Social Computing Systems (2022, background): “accuracy [47, 79], generate realistic action plans that an embodied agent might take [34], or elicit a large variety of commonsense reasoning [48].”