Jailbroken: How Does LLM Safety Training Fail?

By Alexander Wei, Nika Haghtalab, Jacob Steinhardt. Neural Information Processing Systems, 2023

2,019 citations recorded by the source corpus.

Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of"jailbreak"attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes.

Subfields: Artificial Intelligence, Health Informatics, Safety Research

View Paper: https://www.semanticscholar.org/paper/929305892d4ddae575a0fc23227a8139f7681632

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Percy Liang
  3. Yejin Choi
  4. Sergey Levine
  5. Luke S. Zettlemoyer
  6. Jianfeng Gao
  7. Yoshua Bengio
  8. Noah A. Smith

In Health Informatics

  1. Yanzhao Wu
  2. Nael B. Abu-Ghazaleh
  3. Prateek Mittal
  4. George Pappas
  5. Neil Zhenqiang Gong
  6. Michael Backes
  7. Peter Hase
  8. Badhan Chandra Das

In Safety Research

  1. Yang Zhang
  2. Michael Backes
  3. Neil Zhenqiang Gong
  4. Lichao Sun
  5. Yanzhao Wu
  6. Nael B. Abu-Ghazaleh
  7. Dylan Hadfield-Menell
  8. Prateek Mittal

Citations in the corpus