The Capacity for Moral Self-Correction in Large Language Models

By Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamil.e Lukovsiut.e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez and 39 more. 2023

214 citations recorded by the source corpus.

We test the hypothesis that language models trained with reinforcement learning from human feedback (RLHF) have the capability to"morally self-correct"-- to avoid producing harmful outputs -- if instructed to do so. We find strong evidence in support of this hypothesis across three different experiments, each of which reveal different facets of moral self-correction. We find that the capability for moral self-correction emerges at 22B model parameters, and typically improves with increasing model size and RLHF training. We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination. As such, they can follow instructions to avoid certain kinds of morally harmful outputs. We believe our results are cause for cautious optimism regarding the ability to train language models to abide by ethical principles.

Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Yejin Choi
  5. Hannaneh Hajishirzi
  6. Percy Liang
  7. Quoc V. Le
  8. Noah A. Smith

In Natural Language Processing

  1. Mohit Bansal
  2. Yejin Choi
  3. Luke S. Zettlemoyer
  4. Hannaneh Hajishirzi
  5. Samuel R. Bowman
  6. Noah A. Smith
  7. Jason Aaron Edward Weston
  8. Jianfeng Gao

In Alignment & RLHF

  1. Xinyun Chen
  2. Mari Ostendorf
  3. Noah A. Smith
  4. Maxwell Lin
  5. Swaroop Mishra
  6. Nathanael Scharli
  7. Prithviraj Ammanabrolu
  8. Yushi Hu

Citations in the corpus