Adversarial Training for High-Stakes Reliability

By Daniel M. Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein-Raun, Daniel Haas and 2 more. 2022

79 citations recorded by the source corpus.

In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses an adversary to generate examples to train on in order to achieve better worst-case performance. In this work, we used a safe language generation task (``avoid injuries'') as a testbed for achieving high reliability through adversarial training. We created a series of adversarial training techniques -- including a tool that assists human adversaries -- to find and eliminate failures in a classifier that filters text completions suggested by a generator. In our task, we determined that we can set very conservative classifier thresholds without significantly impacting the quality of the filtered outputs. We found that adversarial training increased robustness to the adversarial attacks that we trained on -- doubling the time for our contractors to find adversarial examples both with our tool (from 13 to 26 minutes) and without (from 20 to 44 minutes) -- without affecting in-distribution performance. We hope to see further work in the high-stakes reliability setting, including more powerful tools for enhancing human adversaries and better ways to measure high levels of reliability, until we can confidently rule out the possibility of catastrophic deployment-time failures of powerful models.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Others use humans in the loop to continuously and dynamically build, break, and ﬁx [20] models in order to continuously make them more robust to failure modes [40, 32, 55, 61].”
The alignment problem from a deep learning perspective (2022, background): “Ziegler et al. [2022] use human-generated examples to drive the probability of unwanted language output extremely low, while Perez et al. [2022a] automate the generation of such examples, as proposed by Christiano [2019c].”
Auditing large language models: a three-layered approach (2023, background): “The case for independent audits rests not only on concerns about the misaligned incentives that technology providers may face but also on concerns about the rapidly increasing capabilities of LLMs [281].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “, 2023], adversarial [Ziegler et al., 2022], and which the reward model is uncertain about [Christiano et al.”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, unknown): “Helping find new jailbreaks/red-teaming models (Ziegler et al., 2022).”
Robust Feature-Level Adversaries are Interpretability Tools (2021, background): “Each of the 11 proposals for building safe AI outlined in [26] explicitly call for adversarial robustness and/or interpretability tools, and recent work from [78] on high-stakes reliability in AI found that interpretability tools strengthened their ability to produce inputs for adversarial training.”