Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

By Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse and 27 more. 2022

837 citations recorded by the source corpus.

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

Subfields: Artificial Intelligence, Sociology and Political Science, Alignment & RLHF, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Quoc V. Le
  5. Jianfeng Gao
  6. Graham Neubig
  7. Yejin Choi
  8. Noah A. Smith

In Sociology and Political Science

  1. Adina Williams
  2. Diyi Yang
  3. John Wieting
  4. Douwe Kiela
  5. Melanie Kambadur
  6. Martin Wattenberg
  7. Eric Michael Smith
  8. Lichao Sun

In Alignment & RLHF

  1. Danqi Chen
  2. Ethan Perez
  3. Jacob Steinhardt
  4. Mengzhou Xia
  5. Alexander Wei
  6. Nika Haghtalab
  7. Amanda Askell
  8. Dan Jurafsky

In Safety, Robustness & Interpretability

  1. Adina Williams
  2. Eric Michael Smith
  3. Percy Liang
  4. Douwe Kiela
  5. Dirk Hovy
  6. James Y. Zou
  7. Dan Jurafsky
  8. Melanie Kambadur

Citations in the corpus