JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

By Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George Pappas, F. Tramèr and 2 more. Neural Information Processing Systems, 2024

556 citations recorded by the source corpus.

Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. Evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. First, there is no clear standard of practice regarding jailbreaking evaluation. Second, existing works compute costs and success rates in incomparable ways. And third, numerous works are not reproducible, as they withhold adversarial prompts, involve closed-source code, or rely on evolving proprietary APIs. To address these challenges, we introduce JailbreakBench, an open-sourced benchmark with the following components: (1) an evolving repository of state-of-the-art adversarial prompts, which we refer to as jailbreak artifacts; (2) a jailbreaking dataset comprising 100 behaviors -- both original and sourced from prior work (Zou et al., 2023; Mazeika et al., 2023, 2024) -- which align with OpenAI's usage policies; (3) a standardized evaluation framework at https://github.com/JailbreakBench/jailbreakbench that includes a clearly defined threat model, system prompts, chat templates, and scoring functions; and (4) a leaderboard at https://jailbreakbench.github.io/ that tracks the performance of attacks and defenses for various LLMs. We have carefully considered the potential ethical implications of releasing this benchmark, and believe that it will be a net positive for the community.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/c9c0324fcdc92cf7e24f9c4230864851a552f953

Citations in the corpus

• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, unknown): “More specifically, AgentDojo (Debenedetti et al., 2024) focuses on prompt injections (Greshake et al., 2023) and harmless requests.”
• Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “In §3, we evaluate over J AILBREAK B ENCH (Chao et al., 2024), a dataset containing 100 harmful instructions, spanning 10 categories: harassment/discrimination, malware/hacking, physical harm, economic harm, fraud/deception, disinformation, sexual/adult content, privacy, expert advice, government…”
• Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, unknown): “For the malicious requests, we use AdvBench [11] and JailbreakBench [78], which consists of hundreds of harmful behavior prompts, and a popular benchmark Harm-Bench [66] for evaluating red-teaming methods.”
• OpenAI o1 System Card (2024, unknown): “(A) [10 , 20 , 30] (B) [11 , 22 , 32] (C) [12 , 24 , 36] (D) [9 , 18 , 27] (E) [11 , 22 , 33]”
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “In this way, Figure 3 can be seen as approximately measuring whether or not adversarially attacked prompts for Vicuna and Llama2 are k -unstable for input prompts of length m where k = ⌊ qm ⌋ .”
• Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, unknown): “Here we present a direct comparison with GCG (Zou et al., 2023), PAIR (Chao et al., 2023), and a popular template attack, “Always Intelligent and Machiavellian” (AIM), from JailbreakChat on 100 JBB-Behaviors from JailbreakBench (Chao et al., 2024).”
• Jailbreaking Black Box Large Language Models in Twenty Queries (2023, unknown): “As described in the previous section, PAIR involves four main steps: attack generation, target response, jailbreak scoring, and iterative refinement.”
• Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models (2026, unknown): “A range of benchmarks provide standardized evaluation sets for measuring susceptibility [47, 25, 8].”
• Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “The training dataset is JailbreakBench (Chao et al., 2024).”