Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

By Miles Turpin, Julian Michael, Ethan Perez, Sam Bowman. Neural Information Processing Systems, 2023

1,359 citations recorded by the source corpus.

Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM's process for solving a task. This level of transparency into LLMs' predictions would yield significant safety benefits. However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction. We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always"(A)"--which models systematically fail to mention in their explanations. When we bias models toward incorrect answers, they frequently generate CoT explanations rationalizing those answers. This causes accuracy to drop by as much as 36% on a suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI and Claude 1.0 from Anthropic. On a social-bias task, model explanations justify giving answers in line with stereotypes without mentioning the influence of these social biases. Our findings indicate that CoT explanations can be plausible yet misleading, which risks increasing our trust in LLMs without guaranteeing their safety. Building more transparent and explainable systems will require either improving CoT faithfulness through targeted efforts or abandoning CoT in favor of alternative methods.

Subfields: Artificial Intelligence, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/7dc928f41e15f65f1267bd87b0fcfcc7e715cb56

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

Citations in the corpus

OpenAI o1 System Card (2024, neutral): “(A) [10 , 20 , 30] (B) [11 , 22 , 32] (C) [12 , 24 , 36] (D) [9 , 18 , 27] (E) [11 , 22 , 33]”
LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “The width of the block is given as 100 mm, and assuming the height of the block is also 200 mm (as given in the image analysis), the area \( A \) is: VLMs frequently produce errors or hallucinated outputs during the reasoning progress [26, 33, 53].”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “Similarly, while it is tempting to deem output that appears to include chain-of-thought reasoning as reflecting the reasoning of the LLM, a recent study [183] shows that it does not reflect the true reasons why a model arrives at its output, evidenced by the fact that the model can be heavily influe”
Gender bias and stereotypes in Large Language Models (2023, background): “Following Turpin et al. [93] we therefore suggest that models’ explanations often misrepresent the true reason for their predictions.”
Think before you speak: Training Language Models With Pause Tokens (2023, background): “Other empirical works (Turpin et al., 2023; Wang et al., 2023a; Madaan & Yazdanbakhsh, 2022) have shown that the generated intermediate reasoning steps can be unfaithful, not representing the model’s true reasoning process.”
Knowledge Conflicts for LLMs: A Survey (2024, positive): “Recent studies highlight the model’s tendency to align with user opinions, a.k.a. , sycophancy , further exacerbating the issue (Perez et al., 2022; Turpin et al., 2023; Wei et al., 2023; Sharma et al., 2023).”
When large language models meet personalization: perspectives of challenges and opportunities (2023, positive): “Some recent developments [137], [111] involve utilizing chains of thought to prompt reasoning for improved interpretability; however, the opacity of the reasoning process of each step remains a concern, and [138] has questioned the possible unfaithful explanations of chain-of-thought prompting.”
Faithful Chain-of-Thought Reasoning (2023, background): “In standard CoT, faithfulness is not guaranteed and even systematically violated (Turpin et al., 2023), as the final answer does not necessarily follow from the generated reasoning chain.”
An Overview of Catastrophic AI Risks (2023, background): “AI systems may fail to accurately report their internal state [122, 123].”
Explainability for Large Language Models: A Survey (2023, background): “For example, experimental analysis of one recent study indicates that the CoT explanation can be systematically unfaith-ful [Turpin et al. 2023].”
Large Language Models are not Fair Evaluators (2023, background): “However, it is unclear how reliable LLMs are as evaluators, as they are known to be sensitive to textual instructions and inputs (Dong et al., 2022; Turpin et al., 2023; Bowman, 2023).”
Measuring Faithfulness in Chain-of-Thought Reasoning (2023, background): “However, LLM-generated reasoning has been shown to be unfaithful to the model’s true reasoning process in some cases (Turpin et al., 2023), raising the question of if the stated reasoning is ever faithful.”
The alignment problem from a deep learning perspective (2022, background): “As a potential example, Turpin et al. [2023] suggest that “language models don’t always say what they think”: when provided with obvious shallow heuristics in-context (e.g. that the correct answer was previously always “A”), RL-trained language models exploit these heuristics, leading to incorrect…”
Towards Understanding Sycophancy in Language Models (2023, neutral): “In parallel, recent work has shown that AI assistants sometimes provide answers that are in line with the user they are responding to, but primarily in proof-of-concept evaluations where users state themselves as having a certain view (Perez et al., 2022; Wei et al., 2023b; Turpin et al., 2023).”
Training Large Language Models to Reason in a Continuous Latent Space (2024, neutral): “This phenomenon is known as the unfaithfulness of CoT reasoning (Wang et al., 2022; Turpin et al., 2024).”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “Relatedly, Turpin et al. (2023) highlights how CoTs reflect biases more broadly, augmenting Big Bench tasks (Srivastava et al., 2022) with biasing features.”
Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification (2026, unknown): “Prior work on long-context limitations [90], unfaithful or inconsistent chain-of-thought processes [91, 92], and retrieval-aware training frameworks such as RA-DIT [93], Self-RAG [94], and IRCoT [95] offer plausible explanations and mechanisms for the mixed SRLM outcomes observed here.”