Red Teaming Language Models with Language Models

By Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, Geoffrey Irving. Conference on Empirical Methods in Natural Language Processing, 2022

1,219 citations recorded by the source corpus.

Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases (“red teaming”) using another LM. We evaluate the target LM’s replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot’s own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is one promising tool (among many needed) for finding and fixing diverse, undesirable LM behaviors before impacting users.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

View Paper: https://arxiv.org/abs/2202.03286

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, background): “This should make it easier to scale up automated red teaming [Perez et al., 2022] in future work, since training intensively for harmlessness would otherwise result in a model that simply refuses to be helpful.”
Emergent Abilities of Large Language Models (2022, neutral): “Overall, emergent abilities should probably be viewed as a function of many correlated variables.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, neutral): “, 2022) or auxiliary model-based feedback (Perez et al., 2022).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “One potentially useful tool for addressing harm is red teaming—using manual or automated methods to adversarially probe a language model for harmful outputs, and then updating the model to avoid such outputs [42, 20, 3, 11].”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…measurement templates, which form the basis for prompts that can be used to measure bias (Rudinger et al., 2018; May et al., 2019; Sheng et al., 2019; Kurita et al., 2019; Webster et al., 2020; Gehman et al., 2020; Huang et al., 2020; Vig et al., 2020; Kirk et al., 2021; Perez et al., 2022).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “…and mitigate unsafe behaviour of conversational models (e.g. Dinan et al., 2019; Xu et al., 2021; Ouyang et al., 2022; Thoppilan et al., 2022; Perez et al., 2022; Dinan et al., 2022), this work aims to provide a framework to help practitioners think through the conflicts and tensions that…”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “To train more robustly against these violations, we seed dialogues with LM-generated questions designed to elicit bad behaviour following Perez et al. (2022).”
Self-critiquing models for assisting human evaluators (2022, background): “[ZNC22] and [PHS22] use model assistance to find adversarial examples for language model classifications and generations, respectively.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “…Jaques et al. (2017b, 2019) and Ziegler et al. (2019) or more recently Stiennon et al. (2020), Ouyang et al. (2022), Bai et al. (2022) and Perez et al. (2022) have applied variations of KL-control (Todorov, 2007; Kappen et al., 2012) which adds a penalty term to the reward term so that…”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “, 2020; Smith and Williams, 2021) or several axes of demographic axis more broadly (Barikeri et al., 2021; Perez et al., 2022; Smith et al., 2022).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, methodology): “…2019; Bang et al., 2021; Nadeem et al., 2021), extracted real-world data (Gehman et al., 2020; Sheng et al., 2021; Schick et al., 2021; Deng et al., 2022; Zhou et al., 2022), crowdsourcing workers (Xu et al., 2020), or automatic generation based on pretrained language models (Perez et al., 2022).”
Adversarial Training for High-Stakes Reliability (2022, background): “There is a large and growing literature on both adversarial attacks and adversarial training for large language models [33, 34, 35, 36, 37].”
Predictability and Surprise in Large Generative Models (2022, methodology): “Finally, we should consider how to augment (or complement) manual red-teaming with automated methods [51].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[81] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.”
StarCoder: may the source be with you! (2023, background): “While API access allows researchers to experiment with these models, it limits their ability to research LLM safety (Perez et al., 2022), inspect the models’ inner workings (Olsson et al., 2022), and contribute to model improvements (Togelius & Yannakakis, 2023).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…learning: up-dating models trained on static text using super-vision either from humans (Stiennon et al., 2020; Ouyang et al., 2022; Scheurer et al., 2022) or from proxies for human behavior or social environments (Ammanabrolu et al., 2022a,b) based on broad coverage LLMs (Perez et al., 2022).”