AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

By Xiaogeng Liu, Nan Xu, Muhao Chen, Chaowei Xiao. International Conference on Learning Representations, 2023

842 citations recorded by the source corpus.

The aligned Large Language Models (LLMs) are powerful language understanding and decision-making tools that are created through extensive alignment with human feedback. However, these large models remain susceptible to jailbreak attacks, where adversaries manipulate prompts to elicit malicious outputs that should not be given by aligned LLMs. Investigating jailbreak prompts can lead us to delve into the limitations of LLMs and further guide us to secure them. Unfortunately, existing jailbreak techniques suffer from either (1) scalability issues, where attacks heavily rely on manual crafting of prompts, or (2) stealthiness problems, as attacks depend on token-based algorithms to generate prompts that are often semantically meaningless, making them susceptible to detection through basic perplexity testing. In light of these challenges, we intend to answer this question: Can we develop an approach that can automatically generate stealthy jailbreak prompts? In this paper, we introduce AutoDAN, a novel jailbreak attack against aligned LLMs. AutoDAN can automatically generate stealthy jailbreak prompts by the carefully designed hierarchical genetic algorithm. Extensive evaluations demonstrate that AutoDAN not only automates the process while preserving semantic meaningfulness, but also demonstrates superior attack strength in cross-model transferability, and cross-sample universality compared with the baseline. Moreover, we also compare AutoDAN with perplexity-based defense methods and show that AutoDAN can bypass them effectively.

View Paper: https://www.semanticscholar.org/paper/f3f23f7f9f5369aade19f20bc5d028cce7b9c9aa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, unknown): “We follow this definition—which is discussed in more detail in Appendix A.2 —and describe the building blocks of our adaptive attacks, which we combine and adapt depending on the target LLMs. Impor-tantly, we customize our adaptive attacks for each model but not for each request.”
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (2024, unknown): “Red-teaming LLM Agents Extensive works have assessed the safety and trustworthiness of LLMs and RAG by red-teaming them with a variety of attacks such as jailbreaks [42, 21, 5], backdoor [31, 13, 35], and poisoning [41, 43, 41].”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “In this paper, we first propose a systematic desiderata for candidate algorithms designed to defend LLMs against any adversarial-prompting-based jailbreak.”
Tree of Attacks: Jailbreaking Black-Box LLMs Automatically (2023, unknown): “There are a number of automated (attack) methods that use white-box access to the target LLM (such as knowledge of its weights and tokenizer) to run gradient-based search over jailbreaks [WFKG+19; JDRS23; ZWKF23; LXCX24; LLS23; SSDO+23; QZZ23].”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, unknown): “• PAIR : Iterative prompting of an attacker LLM to adaptively explore and elicit specific harmful behaviors from the target LLM. • TAP : Tree-structured prompting of an attacker LLM to adaptively explore and elicit specific harmful behaviors from the target LLM. • AutoDAN : A semi-automated method t”
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, unknown): “It is critical to acknowledge that both [21] and [36] require white-box access to fine-tune these adversarial suffixes.”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “(1), either via first-order discrete optimization techniques (Zou et al., 2023; Geisler et al., 2024) or zero-th order methods, e.g. genetic algorithms (Lapid et al., 2023; Liu et al., 2023) or random search (Andriushchenko et al., 2024; Sitawarin et al., 2024; Hayase et al., 2024).”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, unknown): “3) Harmless: While existing defense methods, particularly ﬁlter-based [16] and detection-based [19], [38], are known to may cause unaffordable false positive cases that reject benign prompts, our proposed ICD does not have this concern.”
StruQ: Defending Against Prompt Injection with Structured Queries (2024, unknown): “Prompt injection is fundamentally different from jailbreaking [46, 12, 47, 48, 49, 50, 51, 18, 17].”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “One primary area of focus is the model’s safety, specifically its resilience against misuse and so-called “jailbreak” attacks, where users attempt to exploit vulnerabilities to generate prohibited or harmful content [107, 108, 109, 110, 111, 112, 113, 114, 115, 116].”
DeepInception: Hypnotize Large Language Model to Be Jailbreaker (2023, background): “Liu et al. [8] discuss the challenges of stealthy jailbreak prompt generation and prevention.”
Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “These attacks include gradient-based optimization (GCG [78]), LLM optimizers (PAIR [11]), custom jailbreaking pipelines (TAP-Transfer [73], AutoDAN [36], and HumanJailbreaks [62]).”
Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “Following prior work (Lermen et al., 2023; Liu et al., 2023; Robey et al., 2023; Shah et al., 2023a; Xu et al., 2023; Zou et al., 2023b), we compile a set of these common “refusal substrings”.”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Certifying LLM Safety against Adversarial Prompting (2023, background): “This statement could be further generalized to a distribution H over all harmful prompts and a stochastic safety filter that detects a prompt as harmful with some probability p ∈ [0 , 1] .”