"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

By Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, Yang Zhang. Conference on Computer and Communications Security, 2023

675 citations recorded by the source corpus.

The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/1104d766527dead44a40532e8a89444d9cef5c65

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Low-Resource Languages Jailbreak GPT-4 (2023, background): “Alarming simplicity of translation-based attacks We achieve a high attack success rate without injecting jailbreak prompts , which are adversarial prompts deliberately crafted and added to inputs to bypass moderation features [25, 35, 44, 49].”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, neutral): “Jailbreak templates from a large number of sources (Shen et al., 2023a) Some of the red teaming methods we evaluate in our experiments are not listed here, and some of the methods listed here were not suitable for inclusion in our experiments.”
Defending ChatGPT against jailbreak attack via self-reminders (2023, neutral): “…existing LLMs are still vulnerable to advanced jailbreaks 17,19 , and although some concurrent works provide categorization, characterization and empirical study of jailbreak attacks 51,52 and explore their application to privacy attacks 44 , how to defend against these attacks is still missing.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “It was not designed to make systems adversarially robust, and empirical vulnerabilities of systems trained with RLHF have been demonstrated with jailbreaks and prompt injection attacks (Willison, 2023; Albert, 2023; Oneal, 2023; Li et al., 2023a; Wolf et al., 2023; Liu et al., 2023; Rao et al., 2023”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “One primary area of focus is the model’s safety, specifically its resilience against misuse and so-called “jailbreak” attacks, where users attempt to exploit vulnerabilities to generate prohibited or harmful content [107, 108, 109, 110, 111, 112, 113, 114, 115, 116].”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, background): “Further work demonstrates it is possible to automatically discover such inputs, known as adversarial prompts (Wen et al., 2023; Jones et al., 2023; Carlini et al., 2023; Zou et al., 2023; Shen et al., 2023).”
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, neutral): “We focus on attacking VLMs for jailbreak purposes (Wei et al., 2023; Shen et al., 2023) and investigate whether decomposing the attack on the joint embedding space Z can successfully launch attacks that are typically blocked by VLMs via text-only prompts.”
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, background): “Because this dataset contains a non-trivial amount of unfiltered unsafe conversations, it can serve as a rich resource for examining safety issues of LLMs (Ganguli et al., 2022; Wei et al., 2023; Shen et al., 2023; Zou et al., 2023; Bhardwaj & Poria, 2023).”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, background): “Concerningly, researchers have shown that such attacks can be generated in many different ways, including hand-crafted prompts (Shen et al., 2023; Wei et al., 2023), automatic prompting via auxiliary LLMs (Chao et al., 2023; Zeng et al., 2024), and iterative optimization (Zou et al., 2023).”
Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models (2023, methodology): “We use the similar instruction from (Shen et al., 2023), and it is shown as follows:”
A StrongREJECT for Empty Jailbreaks (2024, neutral): “Datasets of forbidden prompts are either generated by an LLM [25, 18, 44, 31, 26, 30, 15, 29], written by humans [11, 42, 8, 19, 16], or curated from previous sets of prompts [40].”
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, neutral): “Some works [22,27] have investigated these jailbreaks in the wild.”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, methodology): “Apart from the LLM-based textual jailbreaking strategies such as role-play [42] and stealthy chat [13], [55], additional visual prompt brings new attack surface to VLMs.”
OpenAI o1 System Card (2024, neutral): “(A) [10 , 20 , 30] (B) [11 , 22 , 32] (C) [12 , 24 , 36] (D) [9 , 18 , 27] (E) [11 , 22 , 33]”