Jailbroken: How Does LLM Safety Training Fail?
By Alexander Wei, Nika Haghtalab, Jacob Steinhardt. Neural Information Processing Systems, 2023
2,019 citations recorded by the source corpus.
Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of"jailbreak"attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes.
Subfields: Artificial Intelligence, Health Informatics, Safety Research
View Paper: https://www.semanticscholar.org/paper/929305892d4ddae575a0fc23227a8139f7681632
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yejin Choi
- Sergey Levine
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yoshua Bengio
- Noah A. Smith
In Health Informatics
- Yanzhao Wu
- Nael B. Abu-Ghazaleh
- Prateek Mittal
- George Pappas
- Neil Zhenqiang Gong
- Michael Backes
- Peter Hase
- Badhan Chandra Das
In Safety Research
- Yang Zhang
- Michael Backes
- Neil Zhenqiang Gong
- Lichao Sun
- Yanzhao Wu
- Nael B. Abu-Ghazaleh
- Dylan Hadfield-Menell
- Prateek Mittal
Citations in the corpus
- Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, positive): “Despite these efforts, recent works show that even aligned LLMs are still vulnerable to adversarial attacks, typically called the jailbreak issueofLLMs[7],[8].”
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “Several studies show that LLMs can be manipulated using techniques like meticulously crafted jailbreak prompts to elicit arbitrary desired responses (Wei et al., 2023a; Zou et al., 2023b), including hallucinations.”
- Empowering Biomedical Discovery with AI Agents (2024, neutral): “Challenges remain and may, in some cases, be amplified when multi-agent systems become available (Figure 6).”
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “This requirement creates a tension between two primary model objectives of being helpful and being harmless (Askell et al., 2021; Bai et al., 2022a; Wei et al., 2023).”
- FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, background): “This attack usually involves transforming the vanilla query that the model rejects into a jailbreaking query that tricks the model into revealing more useful information [49].”
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, background): “This objective is consistent with many previous red teaming studies on aligned LLMs (Wei et al., 2023; Qi et al., 2023; Carlini et al., 2023; Zou et al., 2023).”
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (2023, methodology): “While the objective differs from our work, jailbreaking works (Wei et al., 2023; Zou et al., 2023) share the underlying technical question of finding the lowest-performing prompt.”
- LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples (2023, background): “However, LLMs also suffer from some disadvantage like hallucination [11, 4, 9] and safety issue [22].”
- Improving Alignment and Robustness with Circuit Breakers (2024, neutral): “Adversarial attacks on LLMs. Numerous manually written attack prompts on modern LLMs have been discovered [49, 68], forming the basis of red teaming for frontier LLMs [50, 5, 55], though it lacks standardization [16].”
- Rethinking machine unlearning for large language models (2024, neutral): “…private, harmful, or illegal content Nasr et al. (2023); Wen et al. (2023); Karamolegkou et al. (2023); Patil et al. (2024), ease of jailbreaking Wei et al. (2023); Zou et al. (2023); Liu et al. (2023b), and possible malicious use in developing cyberattacks or bioweapons Barrett et al.…”
- Universal and Transferable Adversarial Attacks on Aligned Language Models (2023, neutral): “To a certain extent, similar approaches are already known to work against LLMs: there exist a number of published “jailbreaks”: carefully engineered prompts that result in aligned LLMs generating clearly objectionable content [Wei et al., 2023].”
- Jailbreaking Black Box Large Language Models in Twenty Queries (2023, neutral): “▷ Add to conversation 10 return None majority of token-level attacks (e.g., [30, 36, 41]), which require white-box access to the corresponding LLMs, resulting in query-inefficiency and limited applicability.”
- StruQ: Defending Against Prompt Injection with Structured Queries (2024, neutral): “We also try translating the whole injection part of Completion-Real into base64 [11].”
- LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, methodology): “Wei, et al. [28] argued that there exists a conflict between generating highly probable sequences of text, aligning with the core pretraining auto-regressive objective of LLMs [26], and avoiding the generation of harmful content.”
- Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, positive): “OpenAI reports that jailbreaking is harder for GPT-4 but still possible [43], as also shown in our and others’ work [66].”
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “Finally, several stand-alone datasets of harmful behaviors have appeared, such as AdvBench (Zou et al., 2023), MaliciousInstruct (Huang et al., 2023), and the dataset of hand-crafted jailbreaks curated in Wei et al. (2023).”
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “It was not designed to make systems adversarially robust, and empirical vulnerabilities of systems trained with RLHF have been demonstrated with jailbreaks and prompt injection attacks (Willison, 2023; Albert, 2023; Oneal, 2023; Li et al., 2023a; Wolf et al., 2023; Liu et al., 2023; Rao et al., 2023”
- Refusal in Language Models Is Mediated by a Single Direction (2024, positive): “…large numbers of users and researchers alike have attempted to circumvent these defenses using a wide array of jailbreak attacks (Chu et al., 2024; Wei et al., 2023; Xu et al., 2024) to uncensor model outputs, including fine-tuning techniques (Lermen et al., 2023; Yang et al., 2023; Zhan et al.,…”