Towards Understanding Sycophancy in Language Models

By Mrinank Sharma, Meg Tong, Tomasz Korbak, David Kristjanson Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott Johnston and 9 more. 2023

1,082 citations recorded by the source corpus.

Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “In settings where annotators cannot do this, we would expect issues like hallucination (Zhang et al., 2023) and sycophancy (Perez et al., 2022b; Sharma et al., 2023) to be exacerbated.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “This behavior, encouraged by human feedback used in fine-tuning, is observed in state-of-the-art AI assistants across various tasks (Sharma et al., 2023).”
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, neutral): “The dying declaration exception is not just “commonly” invoked in homicide prosecutions; FRE 804(b)(2) specifically limits the admissibility of dying declarations to cases involving “a prosecution for homicide or in a civil case,” not any case.”
Managing extreme AI risks amid rapid progress (2023, background): “These R&D challenges include: Oversight and honesty: More capable AI systems can better exploit weaknesses in technical over-sight and testing 42,46,47 —for example, by producing false but compelling output 43,48,49 .”
Knowledge Conflicts for LLMs: A Survey (2024, positive): “Recent studies highlight the model’s tendency to align with user opinions, a.k.a. , sycophancy , further exacerbating the issue (Perez et al., 2022; Turpin et al., 2023; Wei et al., 2023; Sharma et al., 2023).”
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, methodology): “…to undesirable features in inputs (Min et al., 2022; Webson and Pavlick, 2022; Dasgupta et al., 2022; Parrish et al., 2022; Perez et al., 2022; Sharma et al., 2023), and our results suggest that models’ CoT explanations can serve to rationalize giving answers in line with biases while failing…”