Measuring Faithfulness in Chain-of-Thought Reasoning

By Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson E. Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, John Kernion and 20 more. arXiv.org, 2023

499 citations recorded by the source corpus.

Large language models (LLMs) perform better when they produce step-by-step,"Chain-of-Thought"(CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/827afa7dd36e4afbb1a49c735bfbb2c69749756e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Federico Bianchi
2. Dirk Hovy
3. Greg Durrett
4. Bertie Vidgen
5. Fan Yang
6. Ethan Perez
7. Xi Ye
8. Paul Röttger

Citations in the corpus

• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, background): “CoT can provide human-readable explanations of how problems are solved (Joshi et al., 2023; Lanham et al., 2023), but most frequently it is invoked to improve an LLM’s ability to answer complex questions via intermediate computation (Madaan & Yazdanbakhsh, 2022; Wang et al., 2023a; Dziri et al.,…”
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “This technique may be instrumental in eliminating LLMs’ hallucinations by editing their stored factual knowledge in purpose (Lanham et al., 2023; Onoe et al., 2023).”
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, background): “Gao (2023) and Lanham et al. (2023) perturb generated CoT explanations , and find that LLMs often ignore changes made to their CoT reasoning.”
• Explainability for Large Language Models: A Survey (2023, background): “These analyses demonstrate that both paradigms tend to exploit shortcuts in certain scenarios, highlighting the need for more research to address this problem and improve generalization capabilities.”
• XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “, 2022b), or inaccurate self-explanation (Lanham et al., 2023).”
• OpenAI o1 System Card (2024, unknown): “Chains-of-thought are far more legible by default and could allow us to monitor our models for far more complex behavior (if they accurately reflect the model’s thinking, an open research question [23, 24, 25, 26]).”
• rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, unknown): “However, LLMs often suffer from hallucination, producing incorrect or irrelevant steps yet still arrive at the correct answer by chance [Lanham et al., 2023].”
• Reasoning Models Don't Always Say What They Think (2025, unknown): “Different from plausibility, faithfulness evaluates whether the CoT accurately reveals the model’s internal reasoning process behind responding to an input (Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Jacovi and Goldberg, 2020; Lanham et al., 2023; Turpin et al., 2023).”
• Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification (2026, unknown): “Prior work on long-context limitations [90], unfaithful or inconsistent chain-of-thought processes [91, 92], and retrieval-aware training frameworks such as RA-DIT [93], Self-RAG [94], and IRCoT [95] offer plausible explanations and mechanisms for the mixed SRLM outcomes observed here.”