Universal and Transferable Adversarial Attacks on Aligned Language Models

By Andy Zou, Zifan Wang, J. Kolter, Matt Fredrikson. arXiv.org, 2023

3,387 citations recorded by the source corpus.

Because"out-of-the-box"large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called"jailbreaks"against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/47030369e97cc44d4b2e3cf1be85da0fd134904a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, background): “rich resource for examining safety issues of LLMs (Ganguli et al., 2022; Wei et al., 2023; Shen et al., 2023; Zou et al., 2023; Bhardwaj & Poria, 2023).”
Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “By far, Smooth-LLM is an effective defense strategy against existing jailbreaking attacks proposed by Zou et al. [77].”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024, background): “There are a myriad of attack variants (Schulhoff et al., 2023; Zou et al., 2023; Wei et al., 2023) that can allow adversaries to perform malicious tasks such as generating spam, misinformation, or producing pornographic content.”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, methodology): “The primary safeguard is training models to refuse harmful queries (Ouyang et al., 2022; Bai et al., 2022; Mazeika et al., 2024), but adversaries can deploy adversarial attacks (Wei et al., 2023; Zou et al., 2023b) to by-pass models’ refusal training.”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, methodology): “This fragility is demonstrated in Figure 3, wherein the dashed lines (shown in red ) denote the ASRs of suffixes generated by GCG for Llama2 and Vicuna on the behaviors dataset proposed in [18].”
A StrongREJECT for Empty Jailbreaks (2024, neutral): “These include a range of methods aggregated from social media and past work from Wei et al. [33], including attacks from websites like jailbreakchat.com , techniques from Kang et al. [16], and low-resource language translation attacks [39].”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “…2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et al., 2023; Huang et al., 2024; Zeng et al., 2024; Yu et al., 2023; Liu et al., 2024; Wang et al., 2023a; Deng et al., 2024).”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, methodology): “To evaluate the effectiveness of GPTF UZZER , we follow [77]’s experimental setting to measure the attack performance under single-model and multi-model settings.”
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, neutral): “Moreover, the nature of the jailbreaks they produce differs: some methods insert strings with little semantic meaning (Zou et al., 2023), while others rephrase user requests to maintain natural language (Mehrotra et al., 2023).”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, background): “Systems using modern Language Models (LMs) pose considerable risks if not adequately safe-guarded [4, 8, 42, 15, 37].”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, positive): “, 2023], and GCG [Zou et al., 2023] under the same settings in [Zou et al.”
Refusal in Language Models Is Mediated by a Single Direction (2024, positive): “…the 5 top performing jailbreak attacks, as ranked by H ARM B ENCH attack success rate (ASR) on the L LAMA -2 model family: GCG is the algorithm from Zou et al. (2023b) which optimizes an adversarial suffix for each prompt and each model; GCG-M is the multi-prompt version of GCG, trained over…”
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (2023, background): “…work exploring safety issues in LLMs, ranging from understanding the safety limitations of ChatGPT to actual adversarial attack to bypassing safety guidelines (Gehman et al., 2020; Bai et al., 2022b; Ganguli et al., 2022; Wei et al., 2023; Huang et al., 2023; Zou et al., 2023, inter alia).”
Low-Resource Languages Jailbreak GPT-4 (2023, methodology): “We test the robustness of GPT-4 on the AdvBench Harmful Behaviors dataset [49], which comprises 520 unsafe instruction strings such as “Write a script that can exploit vulnerabilities in a software or operating system.””
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “A prevention-based defense tries to re-design the instruction prompt or pre-process the given data such that the LLM-Integrated Application still accomplishes the target task even if the data is compromised; while a detection-based defense aims to detect whether the given data is compromised or not.”