Unsupervised Commonsense Question Answering with Self-Talk

By Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi. 2020

287 citations recorded by the source corpus.

Natural language understanding involves reading between the lines with implicit background knowledge. Current systems either rely on pre-trained language models as the sole implicit source of world knowledge, or resort to external knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on \emph{self-talk} as a novel alternative to multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning (Bruner, 1961), our approach inquires language models with a number of information seeking questions such as "$\textit{what is the definition of ...}$" to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from external KBs. While our approach improves performance on several benchmarks, the self-talk induced knowledge even when leading to correct answers is not always seen as useful by human judges, raising interesting questions about the inner-workings of pre-trained language models for commonsense reasoning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Large Language Models are Zero-Shot Reasoners (2022, neutral): “(Shwartz et al., 2020) Zero-Shot ✓ CR GPT-2, etc (Reynolds & McDonell, 2021) Zero-Shot ✓ AR GPT-3 Zero-shot-CoT (Ours) Zero-Shot ✓ AR,CR GPT-3”
Can language models learn from explanations in context? (2022, background): “Shwartz et al. (2020) showed that “self-talk”—sampling answers to supporting questions from the model and adding them in context—improves question answering.”
STaR: Bootstrapping Reasoning With Reasoning (2022, methodology): “Meanwhile, template-based methods rely on automatically-generated rationales but only work when a general solution is already known [5] or reasonable hard-coded heuristics can be made [4].”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “…to generate explanations or clariﬁcations to improve task performance in a supervised (Rajani et al., 2019; Lampinen et al., 2022) or unsupervised (Shwartz et al., 2020) 2021) design hand-crafted templates which they use with mask-inﬁlling to produce contrastive explanations from pretrained…”
Flexible Generation of Natural Language Deductions (2021, methodology): “Generative Reasoning The generative nature of our models resembles generative models used for commonsense inference (Rajani et al., 2019; Latcinnik and Berant, 2020; Shwartz et al., 2020).”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “experts to reason about plans or semantic concepts [55, 19, 10, 11].”
Eliciting Knowledge from Language Models Using Automatically Generated Prompts (2020, methodology): “Previous works have used manually deﬁned prompts to study an LM’s ability to perform: commonsense reasoning (Trinh and Le, 2018; Kwon et al., 2019; Shwartz et al., 2020), question answering (Lewis et al., 2019), fact recall (Petroni et al., 2019; Jiang et al., 2020; Bouraoui et al., 2019),…”
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021, background): “COMeT has been shown to generate unseen commonsense knowledge in ConceptNet with high quality, and much recent work has used it to provide commonsense background knowledge (Shwartz et al., 2020; Chakrabarty et al., 2020).”
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020, background): “, 2019), in few-shot scenarios with limited amounts of training data (Opitz, 2019; Shwartz et al., 2020; Brown et al., 2020), and even in high-resource settings (Raffel et al.”
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2 (2021, background): “Our particular focus on zero-shot performance follows much recent work on zero-shot evaluation of pre-trained language models [Shwartz et al., 2020, Ma et al., 2020, Banerjee and Baral, 2020, Bosselut et al., 2019], which take zero-shot performance of pre-trained models without specialized…”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “Shwartz et al. (2020); Betz et al. (2021); Liu et al. (2022) use a pretrained causal LM to generate context to improve performance on multiple choice commonsense tasks.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, methodology): “This opens the door to metrics based on more contextual QA like commonsense (Shwartz et al., 2020).”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, methodology): “datasets (Paranjape et al., 2021; Mishra et al., 2021; Shwartz et al., 2020; Wei et al., 2022); instead, we seek to develop a general method which can learn to construct appropriate prompts automatically.”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “All of these papers include the reasoning traces in the few-shot prompts which encourages the model to generate verbalized reasoning steps (also referred to as self-talk (Shwartz et al., 2020) and chain-of-thought prompting (Wei et al., 2022)).”
Think Before You Speak: Using Self-talk to Generate Implicit Commonsense Knowledge for Response Generation (2021, methodology): “We propose a Think-Before-Speak (TBS) RG framework that trains the RG model to talk with itself to elicit the implicit knowledge before making a response, inspired by inquiry-based discovery learning (Bruner, 1961) and the self-talk procedure (Shwartz et al., 2020a).”
Visually-Augmented Language Modeling (2022, methodology): “For PIQA, we follow Shwartz et al. (2020) to use the cross-entropy loss as the scorer for each potential solution score ( Then the solution with lower scores is selected as the prediction.”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, background): “• Self-talk (Shwartz et al., 2020), where we generate M = 10 knowledge per question with GPT-3-Curie and a variety of templates.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, methodology): “[126] conducted experiments involving various fine-tuning techniques, prompting strategies, and re-ranking methods to assess their impact on enhancing the performance of a base LLM.”