The Capacity for Moral Self-Correction in Large Language Models
By Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamil.e Lukovsiut.e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez and 39 more. 2023
214 citations recorded by the source corpus.
We test the hypothesis that language models trained with reinforcement learning from human feedback (RLHF) have the capability to"morally self-correct"-- to avoid producing harmful outputs -- if instructed to do so. We find strong evidence in support of this hypothesis across three different experiments, each of which reveal different facets of moral self-correction. We find that the capability for moral self-correction emerges at 22B model parameters, and typically improves with increasing model size and RLHF training. We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination. As such, they can follow instructions to avoid certain kinds of morally harmful outputs. We believe our results are cause for cautious optimism regarding the ability to train language models to abide by ethical principles.
Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Hannaneh Hajishirzi
- Percy Liang
- Quoc V. Le
- Noah A. Smith
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Samuel R. Bowman
- Noah A. Smith
- Jason Aaron Edward Weston
- Jianfeng Gao
In Alignment & RLHF
- Xinyun Chen
- Mari Ostendorf
- Noah A. Smith
- Maxwell Lin
- Swaroop Mishra
- Nathanael Scharli
- Prithviraj Ammanabrolu
- Yushi Hu
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “, 2022), and follow-up instructions can be used to refine initial generations to make them more useful, engaging, and unbiased (Ganguli et al., 2023; Madaan et al., 2023).”
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “Incorporating human feedback into the process of language model (LM) training has been shown as effective to reduce false, toxic and other undesired model generation outputs [29, 3, 2, 33, 10].”
- Cognitive Architectures for Language Agents (2023, methodology): “Subsequent work used the LLM itself as a pre-processing step, eliciting targeted reasoning to foreground a particular aspect of the problem (Bai et al., 2022; Jin et al., 2022; Ganguli et al., 2023; Madaan et al., 2023; Saunders et al., 2022; Kim et al., 2023; Kirk et al., 2023) or generate intermed”
- Teaching Large Language Models to Self-Debug (2023, background): “Despite recent promising progress showing that large language models can generate critiques to avoid harmful model outputs (Ganguli et al., 2023; Bai et al., 2022) and improve their performance on some natural language tasks (Shinn et al.”
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (2023, background): “[14] to demonstrate the moral self-correction in Dromedary without additional instructions.”
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, background): “Prompting approaches can reduce the sensitivity of CoT explanations to input perturbations and stereotypes (Shaikh et al., 2022; Ganguli et al., 2023; Shi et al., 2023), which our findings on prompting for debiasing corroborate.”
- Low-Resource Languages Jailbreak GPT-4 (2023, background): “We want to highlight that significant efforts have been invested to design better safeguards against jailbreak prompts and make the engineering of adversarial prompts harder [14, 15, 40]; therefore, our attack’s success in the absence of a jailbreak prompt is particularly alarming.”
- Defending ChatGPT against jailbreak attack via self-reminders (2023, unknown): “Recent studies have explored the capacity of LLMs to validate and correct their own claims 31–33 .”
- The Claude 3 Model Family: Opus, Sonnet, Haiku (methodology): “…in Evaluating AI Systems [10], Red Teaming Language Models to Reduce Harms [11], Capacity for Moral Self-Correction in Large Language Models [12], Towards Measuring the Representation of Subjective Global Opinions in Language Models [13], Frontier Threats Red Teaming for AI Safety [14], and…”
- Inverse Scaling: When Bigger Isn't Better (2023, background): “Ganguli et al. (2023) showed that inverse scaling trends related to bias against demographic groups could be reversed, by having LMs generate text that actively mitigates their biases before answering a question.”
- Generative AI (2023, unknown): “Some scholars even attest to models certain moral self-correcting capabilities (Ganguli et al. 2023), which may attenuate concerns of embedded biases and result in more fairness.”
- Large Language Models as Optimizers (2023, background): “…with natural language feedback to revise the model output, which has shown effectiveness in reducing harmful LLM outputs (Bai et al., 2022; Ganguli et al., 2023), improving reasoning (Shinn et al., 2023; Madaan et al., 2023) and code generation performance (Chen et al., 2023e; Olausson et…”
- Language Models can Solve Computer Tasks (2023, methodology): “1 RCI Prompting The self-critiquing ability of LLMs has demonstrated that LLMs can find errors in their own output by themselves [43, 16].”
- A Comprehensive Overview of Large Language Models (2023, unknown): “Aligning with Prompts: LLMs can be steered with prompts to generate desirable responses without training [156], [157].”
- “According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, background): “First, larger LLMs can be more effectively guided using natural language prompts (Ouyang et al., 2022; Wan et al., 2023; Ganguli et al., 2023).”
- Large Language Models Cannot Self-Correct Reasoning Yet (2023, background): “Self-correction can be effectively employed to make responses align with specific preferences, such as altering the style of responses or enhancing their safety (Bai et al., 2022; Ganguli et al., 2023; Madaan et al., 2023).”