The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning

By Xi Ye, Greg Durrett. 2022

259 citations recorded by the source corpus.

Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially. We further show that explanations generated by the LLMs may not entail the models' predictions nor be factually grounded in the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs' predictions post-hoc. Through analysis in our three settings, we show that explanations judged by humans to be good--logically consistent with the input and the prediction--more likely cooccur with accurate predictions. Following these observations, we train calibrators using automatically extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoc across all of our datasets.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Denny Zhou
2. Quoc V. Le
3. Mohit Bansal
4. Matt Gardner
5. Ana Marasovic
6. Jason Wei
7. Percy Liang
8. Graham Neubig

In Prompting & Reasoning

1. Denny Zhou
2. Jie Huang
3. Xuezhi Wang
4. Jason Wei
5. Ed H. Chi
6. Kevin Chen-Chuan Chang
7. Quoc V. Le
8. Wenhu Chen

Citations in the corpus

• Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “In additional experiments, we show self-consistency can robustly boost performance on NLP tasks where adding a chain-of-thought might hurt performance compared to standard prompting (Ye & Durrett, 2022).”
• Towards Reasoning in Large Language Models: A Survey (2022, background): “Ensuring that the ratio-nales produced by LLMs are valid is critical, as incorrect rationales can lead to incorrect final predictions (Ye and Durrett, 2022).”
• Steering Language Models With Activation Engineering (2023, background): “Prompt engineering is the most obvious way to steer a model, but prompting has limited reliability (Ye & Durrett, 2022; Wang et al., 2024).”
• Explainability for Large Language Models: A Survey (2023, background): “The article reveals limitations of current methods and that optimizing human preferences like plausibility may be insufficient for improving counterfactual simulatability.”
• CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, background): “5 https://twitter.com/ben_bogin/status/ 1532022804886978568 6 Another work shows limitations of prompting with explanations (Ye and Durrett, 2022).”