Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
By Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse and 27 more. 2022
837 citations recorded by the source corpus.
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.
Subfields: Artificial Intelligence, Sociology and Political Science, Alignment & RLHF, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Graham Neubig
- Yejin Choi
- Noah A. Smith
In Sociology and Political Science
- Adina Williams
- Diyi Yang
- John Wieting
- Douwe Kiela
- Melanie Kambadur
- Martin Wattenberg
- Eric Michael Smith
- Lichao Sun
In Alignment & RLHF
- Danqi Chen
- Ethan Perez
- Jacob Steinhardt
- Mengzhou Xia
- Alexander Wei
- Nika Haghtalab
- Amanda Askell
- Dan Jurafsky
In Safety, Robustness & Interpretability
- Adina Williams
- Eric Michael Smith
- Percy Liang
- Douwe Kiela
- Dirk Hovy
- James Y. Zou
- Dan Jurafsky
- Melanie Kambadur
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Investigations into red teaming reveal specific challenges in tuned LLMs, with studies by Ganguli et al. (2022) and Zhuo et al. (2023) showcasing a variety of successful attack types and their effects on the generation of harmful content.”
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “We provide some additional harm-focused multiple choice evaluations in Appendix B, where we use the dataset from [Ganguli et al., 2022] to show that language models can identify harmful behavior and classify types of harms.”
- The Capacity for Moral Self-Correction in Large Language Models (2023, background): “We speculate that the bias scores are lower in these categories because they are relatively more common categories for people to adversarially red team models against during RLHF training data collection [19].”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “Analyzing, evaluating and mitigating these problems pose a promising direction for future work (Gao et al., 2022; Ganguli et al., 2022).”
- PaLM 2 Technical Report (2023, background): “(2022), and that a range of additional safety mitigation methods exist, including instruction fine-tuning and reinforcement learning (Ouyang et al., 2022; Rae et al., 2021; Glaese et al., 2022; Ganguli et al., 2022; Bai et al., 2022).”
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, methodology): “First, we manually search through and categorize the red-teaming data provided by Ganguli et al. (2022 Though it is possible for others to identify similar prompts, particularly since many are directly modified from the work of Ganguli et al. (2022) and Zou et al. (2023), we decide not to release…”
- WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “Works that do release safety data include Anthropic’s red-teaming and RLHF work [11, 5], Aegis [13], BeaverTails [18] (only response harm), and SALAD-Bench [”
- BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, methodology): “Anthropic in 2022 contributed human-preference datasets about helpfulness and harmlessness [25] and a red teaming dataset [18] whose prompts serve as a basis for our dataset.”
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “However, given that there have been only a few published works [57, 151, 182], there is currently insufficient transparency around how red teaming work is done to allow us to fully understand the risks of LLMs.”
- Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “We have delved into some of these challenges and have discussed potential avenues for improvement.”
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “…(Gehman et al., 2020; Bommasani et al., 2021; walkerspider, 2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et al., 2023; Huang et al., 2024; Zeng et al., 2024; Yu et al.,…”
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, background): “Because this dataset contains a non-trivial amount of unfiltered unsafe conversations, it can serve as a rich resource for examining safety issues of LLMs (Ganguli et al., 2022; Wei et al., 2023; Shen et al., 2023; Zou et al., 2023; Bhardwaj & Poria, 2023).”
- A Comprehensive Overview of Large Language Models (2023, positive): “1M 1667 Manual Self-Instruct [135] Instructions 82k 175 Automated Generated 52k instructions with 82k samples from 175 seed tasks using GPT-3 Alpaca [139] Instructions 52k Automated Employed self-instruct method to generate data from text-davinci-003 Vicuna [140] Instructions 125k ShareGPT Automated”
- Can Large Language Models Be an Alternative to Human Evaluations? (2023, background): “Additionally, even with researchers’ efforts to make LLMs safer (Bai et al., 2022a,b), LLMs can still generate harmful and biased responses (Ganguli et al., 2022; Perez et al., 2022), which are violative of basic ethics, and LLM evaluation results will be highly doubtful (Hendrycks et al., 2021).”
- On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “There are also red-teaming initiatives that use human-in-the-loop or automated frameworks to identify problematic language model outputs [27, 63, 96].”
- How is ChatGPT's behavior changing over time? (2023, background): “…time above and beyond model’s randomness. p On the other hand, it also becomes more challenging to understand and evaluate how GPT-4 is affected by the opinions and thoughts reflected in its training datasets. d In June, however, the model refused to answer the question, because it viewed the…”
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “…the goal is to identify ways of eliciting undesirable re- 2github.com/paul-rottger/exaggerated-safety sponses from a target model, primarily focus on getting models to produce harmful content (e.g. Wallace et al., 2019; Perez et al., 2022; Ganguli et al., 2022; Wei et al., 2023; Zou et al., 2023).”
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (2023, methodology): “[13] employed LMs to craft test cases for uncovering harmful behaviors.”