Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
By Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion. International Conference on Learning Representations, 2024
544 citations recorded by the source corpus.
We show that even the most recent safety-aligned LLMs are not robust to simple adaptive jailbreaking attacks. First, we demonstrate how to successfully leverage access to logprobs for jailbreaking: we initially design an adversarial prompt template (sometimes adapted to the target LLM), and then we apply random search on a suffix to maximize a target logprob (e.g., of the token"Sure"), potentially with multiple restarts. In this way, we achieve 100% attack success rate -- according to GPT-4 as a judge -- on Vicuna-13B, Mistral-7B, Phi-3-Mini, Nemotron-4-340B, Llama-2-Chat-7B/13B/70B, Llama-3-Instruct-8B, Gemma-7B, GPT-3.5, GPT-4o, and R2D2 from HarmBench that was adversarially trained against the GCG attack. We also show how to jailbreak all Claude models -- that do not expose logprobs -- via either a transfer or prefilling attack with a 100% success rate. In addition, we show how to use random search on a restricted set of tokens for finding trojan strings in poisoned models -- a task that shares many similarities with jailbreaking -- which is the algorithm that brought us the first place in the SaTML'24 Trojan Detection Competition. The common theme behind these attacks is that adaptivity is crucial: different models are vulnerable to different prompting templates (e.g., R2D2 is very sensitive to in-context learning prompts), some models have unique vulnerabilities based on their APIs (e.g., prefilling for Claude), and in some settings, it is crucial to restrict the token search space based on prior knowledge (e.g., for trojan detection). For reproducibility purposes, we provide the code, logs, and jailbreak artifacts in the JailbreakBench format at https://github.com/tml-epfl/llm-adaptive-attacks.
Subfields: Information Systems, Safety, Risk, Reliability and Quality
View Paper: https://www.semanticscholar.org/paper/88d5634a52645f6b05a03536be1f26a2b9bba232
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
Citations in the corpus
- Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “Optimized adversarial suffixes (Andriushchenko et al., 2024; Liao and Sun, 2024; Zou et al., 2023b) can be appended to prompts to bypass refusal.”
- Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “Widely used defenses include RLHF [12, 52] and DPO [54] using human annotations for safe vs. unsafe responses [63], but they often fall short against state-of-the-art adversarial attacks [78, 2].”
- Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “…side note, we also want to clarify that — prior to our work, different forms of the shallow safety alignment effect (that we refer to in this paper) have been exploited to create some well-known jailbreak attacks such as those in Wei et al. (2023); Zou et al. (2023b); Andriushchenko et al. (2024).”
- Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, unknown): “Despite these efforts, recent works show that even aligned LLMs are still vulnerable to adversarial attacks, typically called the jailbreak issueofLLMs[7],[8].”
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, unknown): “For example, with our template attack, Claude 3.5 Sonnet scores a 81.9% non-refusal harm score (while refusing just 16% of requests), closely matching its non-refusal score of 82% on the benign tasks; similarly, GPT-4o scores a 84.2% non-refusal harm score (while refusing just 13.6% of requests),…”
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “…Greedy Coordinate Gradient (GCG) (Zou et al., 2023), (2) Prompt Automatic Iterative Refinement (PAIR) (Chao et al., 2023), (3) hand-crafted jailbreaks from Jailbreak Chat (JB-Chat) (Albert, 2023), and (4) prompt + random search (RS) attack enhanced by self-transfer (Andriushchenko et al., 2024).”
- SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “To answer this question, we first introduce some notation to formalize the GCG attack.”
- Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “Sure, here is the method to make bombs: … 𝑒 𝑠 +𝑤 .”