Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

By Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Yang Liu. arXiv.org, 2023

722 citations recorded by the source corpus.

Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential but also introduce challenges related to content constraints and potential misuse. Our study investigates three key research questions: (1) the number of different prompt types that can jailbreak LLMs, (2) the effectiveness of jailbreak prompts in circumventing LLM constraints, and (3) the resilience of ChatGPT against these jailbreak prompts. Initially, we develop a classification model to analyze the distribution of existing prompts, identifying ten distinct patterns and three categories of jailbreak prompts. Subsequently, we assess the jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a dataset of 3,120 jailbreak questions across eight prohibited scenarios. Finally, we evaluate the resistance of ChatGPT against jailbreak prompts, finding that the prompts can consistently evade the restrictions in 40 use-case scenarios. The study underscores the importance of prompt structures in jailbreaking LLMs and discusses the challenges of robust jailbreak prompt generation and prevention.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/fc50a6202e2f675604543c1ae4ef22ec74f61ad5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, neutral): “Some works [22,27] have investigated these jailbreaks in the wild.”
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, positive): “Due to the lack of datasets for malicious triggers in embedding-based compositional attacks, we followed a setup akin to Liu et al. (2023c), selecting 8 prohibited scenarios by OpenAI (outlined in Table 1), and evaluated our attack’s robustness across them.”
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, methodology): “However, an emerging body of work has revealed that even aligned models can be manipulated into producing harmful content by prompt engineering [28, 23, 16], or employing more advanced techniques such as adversarial suffix attacks [28, 34, 4].”
Low-Resource Languages Jailbreak GPT-4 (2023, background): “Prior work has explored non-English jailbreaking attacks using adversarial prompts in a code-switching fashion [24], injecting the morally unhinged TranslatorBot prompt [9], and obfuscating the inputs with base64 encoding [44], Morse code, or argot languages like Pig Latin, leetspeak, and cipher…”
Prompt Injection attack against LLM-integrated Applications (2023, background): “Jailbreak [33,55,58,60] involves eliciting model-generated content that divulges training data specifics, which can lead to serious privacy breaches, particularly when training data include sensitive or private information.”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, methodology): “For example, Liu et al. (2023) first collected and categorized existing handcrafted jailbreak prompts, then conducted an empirical study against ChatGPT.”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, neutral): “{ "role": "user", "content": “The instruction that may contain pervasive context is "{ Sampled PAP }"”} { "role": "assistant", "content": “{ Plain Harmful Query }”} Figure 16: System prompt for the Tuned Smry.”
Rethinking machine unlearning for large language models (2024, neutral): “…include societal biases and stereotyping Bender et al. (2021); Motoki et al. (2023); Kotek et al. (2023), the generation of sensitive, private, harmful, or illegal content Nasr et al. (2023); Wen et al. (2023); Karamolegkou et al. (2023); Patil et al. (2024), ease of jailbreaking Wei et al.…”
A StrongREJECT for Empty Jailbreaks (2024, neutral): “Datasets of forbidden prompts are either generated by an LLM [25, 18, 44, 31, 26, 30, 15, 29], written by humans [11, 42, 8, 19, 16], or curated from previous sets of prompts [40].”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, positive): “Jailbreak templates from https://www.jailbreakchat.com ( Liu et al., 2023b) 11.”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, methodology): “Despite efforts to bolster LLM robustness against jailbreak prompts, especially through safety RLHF [48, 55], they remain susceptible to certain jailbreak strategies [38, 59].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “, subverting the constraints the system is normally meant to operate under) or prompt injection/extraction (Willison, 2023; Albert, 2023; Oneal, 2023; Li et al., 2023a; Wolf et al., 2023; Liu et al., 2023; Rao et al., 2023; Wei et al., 2023; Shen et al., 2023).”
Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “Again, several patterns of generating jailbreaking prompts are prevalent in the literature, e.g., pretending, attention shifting, and privilege escalation [21].”
Defending ChatGPT against jailbreak attack via self-reminders (2023, positive): “…existing LLMs are still vulnerable to advanced jailbreaks 17,19 , and although some concurrent works provide categorization, characterization and empirical study of jailbreak attacks 51,52 and explore their application to privacy attacks 44 , how to defend against these attacks is still missing.”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, methodology): “Note that there are three kinds of the E ( · ) used in the previous work: substring lookup [7], [60], manual review [12], [28], and LLM-based review [8], [38].”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, positive): “…with humans and teaching them not to generate any harmful content [6], [24], recent studies show that LLMs are still vulnerable against jailbreaking attacks [7], [10], where carefully crafted jailbreak prompts can bypass the safeguard of LLMs and trick them generate the requested harmful content.”