Few-Shot Self-Rationalization with Natural Language Prompts

By Ana Marasovic, Iz Beltagy, Doug Downey, Matthew E. Peters. 2021

119 citations recorded by the source corpus.

Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB -- a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2007), natural language inference (Camburu et al., 2018; Marasović et al., 2022), commonsense reasoning (Zelikman et al.”
Can language models learn from explanations in context? (2022, background): “It could also potentially help with user interpretability, though with the caveat that explanations might not be entirely reliable, at least without further refinement (Marasović et al., 2021; Wiegreffe et al., 2021).”
STaR: Bootstrapping Reasoning With Reasoning (2022, background): “[7] Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E Peters.”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, background): “…includes, for example, diverse decoding and ensembling to improve self-consistency (Wang et al., 2022b), more complex and contrasting rationales (Marasović et al., 2021; Wang et al., 2022a), generating sub-questions and answering them sequentially in least-to-most complexity order (Zhou et…”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, background): “…2021; Liu et al., 2021b), in the forms of task descriptions (Mishra et al., 2021; Raffel et al., 2019), answer demonstrations (Brown et al., 2020), targeting formats (Marasovi´c et al., 2021), etc., which can be positioned before (Wei et al., 2022) or even after (Lampinen et al., 2022) the answer.”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, background): “, 2020) has been effective for tasks beyond QA (Bragg et al., 2021; Marasović et al., 2022), this result suggests that UnifiedQA”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “…of work shows that step-by-step reasoning ben-eﬁts LLM performance (Rajani et al., 2019; Shwartz et al., 2020; Nye et al., 2021; Wei et al., 2022; Marasovi´c et al., 2021; Lampinen et al., 2022) and correspondingly, that this performance can be improved with further guidance and tool use (Zhou…”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “…steps (Wei et al., 2022; Zhou et al., 2022), previous work has tried to add hand-written task descriptions (Raffel et al., 2019b) and targeting formats (Marasovi´c et al., 2021) to the prompts, but such human annotation is often costly and expensive at scale (Li & Liang, 2021; Shin et al., 2020).”
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (2022, background): “One appealing way to gain more insight into predictions obtained through in-context learning is to let the language model “explain itself” (Nye et al., 2021; Wei et al., 2022; Chowdhery et al., 2022; Marasović et al., 2022; Lampinen et al., 2022).”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, background): “One appealing way to gain more insight into predictions obtained through in-context learning is to let the language model “explain itself” (Nye et al., 2021; Wei et al., 2022; Chowdhery et al., 2022; Marasović et al., 2022; Lampinen et al., 2022).”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, methodology): “Our study builds on top of prior works that also gen-100 erate free-text explanations using in-context learning with GPT-3 (Marasovi´c et al., 2022; Wiegreffe et al., 2022).”
Successive Prompting for Decomposing Complex Questions (2022, methodology): “In follow-up works (Schick, 2022; Chowdhery et al., 2022; Marasovi´c et al., 2021) prompting techniques have been used as a mechanism to supervise the model decision with few demonstrations as a conditioning context to guide its predictions on an unseen example.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Some prompting techniques [16,47,60,73] have also been developed for more effective or transparent reasoning in NLP.”