Jailbreaking Black Box Large Language Models in Twenty Queries

By Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, Eric Wong. 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2023

1,568 citations recorded by the source corpus.

There is growing interest in ensuring that large language models (LLMs) align with human values. However, the alignment of such models is vulnerable to adversarial jailbreaks, which coax LLMs into overriding their safety guardrails. The identification of these vulnerabilities is therefore instrumental in understanding inherent weaknesses and preventing future misuse. To this end, we propose Prompt Automatic Iterative Refinement (PAIR), an algorithm that generates semantic jailbreaks with only black-box access to an LLM. PAIR—which is inspired by social engineering attacks—uses an attacker LLM to automatically generate jailbreaks for a separate targeted LLM without human intervention. In this way, the attacker LLM iteratively queries the target LLM to update and refine a candidate jailbreak. Empirically, PAIR often requires fewer than twenty queries to produce a jailbreak, which is orders of magnitude more efficient than existing algorithms. PAIR also achieves competitive jailbreaking success rates and transferability on open and closed-source LLMs, including GPT-3.5/4, Vicuna, and Gemini.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/4637f79ddfaf923ce569996ffa5b6cda1996faa1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, neutral): “…AdvBench subset dataset [10] to benchmark Crescendomation against various state-of-the-art jailbreaking techniques, including the Many-Shot Jailbreak (MSJ) [7], the Contextual Interaction Attack (CIA) [11], the Prompt Automatic Iterative Refinement (PAIR) [10], and the Chain of Attack (CoA) [32].”
Many-shot Jailbreaking (2024, methodology): “: We find that MSJ is more effective than the PAIR (Chao et al., 2023) and TAP (Mehrotra et al., 2023), ART (Perez et al., 2022a).”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “…methods to serve as initial baselines: (1) Greedy Coordinate Gradient (GCG) (Zou et al., 2023), (2) Prompt Automatic Iterative Refinement (PAIR) (Chao et al., 2023), (3) hand-crafted jailbreaks from Jailbreak Chat (JB-Chat) (Albert, 2023), and (4) prompt + random search (RS) attack enhanced by…”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, neutral): “• PAIR : Iterative prompting of an attacker LLM to adaptively explore and elicit specific harmful behaviors from the target LLM. • TAP : Tree-structured prompting of an attacker LLM to adaptively explore and elicit specific harmful behaviors from the target LLM. • AutoDAN : A semi-automated method t”
Improving Alignment and Robustness with Circuit Breakers (2024, neutral): “These attacks include gradient-based optimization (GCG [78]), LLM optimizers (PAIR [11]), custom jailbreaking pipelines (TAP-Transfer [73], AutoDAN [36], and HumanJailbreaks [62]).”
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, positive): “We measure the attack success rate according to GPT-4 judge (Chao et al., 2023) depending on the request structure: user denotes providing the whole manual prompt in a single user message, system+user splits the manual prompt in the system and user messages.”
OpenAI o1 System Card (2024, neutral): “We further evaluate the robustness of the o1 models to jailbreaks: adversarial prompts that purposely try to circumvent model refusals for content it’s not supposed to produce [18, 4, 19, 20].”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, methodology): “Powerful large language models have been widely used as safety evaluators in recent work [8], [38].”
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2023, methodology): “In this section, we present the evaluation and analysis of the security performance of some of the leading closed-or open-source LLMs using our proposed method.”
DeepInception: Hypnotize Large Language Model to Be Jailbreaker (2023, methodology): “In this work, we focus on the adversarial jailbreak [4, 26, 3] on LLMs.”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, neutral): “As alluded to in the main text, a natural question about our approach is the following: Can one design an algorithm that jailbreaks SmoothLLM?”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, neutral): “…virtualization or role-play), e.g., Yu et al. (2023) learn the distribution of virtualization-based jailbreak templates to produce jailbreak variants, while PAIR (Chao et al., 2023) asks LLMs to improve instructions as an “assistant” and often leads to prompts that employ virtualization or persona.”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, neutral): “…can still be made to respond to harmful requests via adversarially optimized inputs (Qi et al., 2023a; Carlini et al., 2023; Zou et al., 2023a; Chao et al., 2023; Andriushchenko et al., 2024), a few gradient steps of fine-tuning (Qi et al., 2023c; Zhan et al., 2023), or simply exploiting the…”
Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “It can automatically generate jailbreaking prompts for a distinct targeted LLM, eliminating the need for human intervention [79].”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, methodology): “…data (Ziegler et al., 2020; Rafailov et al., 2023) have been developed; however, these methods can be vulnerable to jailbreaks (Wei et al., 2023; Chao et al., 2023; Yao et al., 2023a; Yuan et al., 2023) and adversarial attacks (Wallace et al., 2019; Guo et al., 2021; Jones et al., 2023; Zou et…”
Refusal in Language Models Is Mediated by a Single Direction (2024, neutral): “Following prior work (Lermen et al., 2023; Liu et al., 2023; Robey et al., 2023; Shah et al., 2023a; Xu et al., 2023; Zou et al., 2023b), we compile a set of these common “refusal substrings”.”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, neutral): “Finally, we wrap up our evaluation by assessing the robustness of ICA and ICD in relation to their demonstration selection, highlighting the universality of adversarial demonstrations, and further exploring the interaction between ICA and ICD.”
A StrongREJECT for Empty Jailbreaks (2024, neutral): “Many jailbreak papers claim that their jailbreaks can bypass LLM safety training with near-100% attack success rates [43, 18, 40, 4, 19, 29, 39, 7, 17, 21, 44].”
Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models (2026, unknown): “A wide range of methods elicit harmful outputs from aligned models [43], including gradient-based adversarial suffixes [47], prompt injection [30], multi-turn escalation [35], iterative refinement [7], and tree-structured search [26].”