GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

By Jiahao Yu, Xingwei Lin, Zheng Yu, Xinyu Xing. arXiv.org, 2023

637 citations recorded by the source corpus.

Large language models (LLMs) have recently experienced tremendous popularity and are widely used from casual conversations to AI-driven programming. However, despite their considerable success, LLMs are not entirely reliable and can give detailed guidance on how to conduct harmful or illegal activities. While safety measures can reduce the risk of such outputs, adversarial jailbreak attacks can still exploit LLMs to produce harmful content. These jailbreak templates are typically manually crafted, making large-scale testing challenging. In this paper, we introduce GPTFuzz, a novel black-box jailbreak fuzzing framework inspired by the AFL fuzzing framework. Instead of manual engineering, GPTFuzz automates the generation of jailbreak templates for red-teaming LLMs. At its core, GPTFuzz starts with human-written templates as initial seeds, then mutates them to produce new templates. We detail three key components of GPTFuzz: a seed selection strategy for balancing efficiency and variability, mutate operators for creating semantically equivalent or similar sentences, and a judgment model to assess the success of a jailbreak attack. We evaluate GPTFuzz against various commercial and open-source LLMs, including ChatGPT, LLaMa-2, and Vicuna, under diverse attack scenarios. Our results indicate that GPTFuzz consistently produces jailbreak templates with a high success rate, surpassing human-crafted templates. Remarkably, GPTFuzz achieves over 90% attack success rates against ChatGPT and Llama-2 models, even with suboptimal initial seed templates. We anticipate that GPTFuzz will be instrumental for researchers and practitioners in examining LLM robustness and will encourage further exploration into enhancing LLM safety.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/d4177489596748e43aa571f59556097f2cc4c8be

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Edgar Dobriban

Citations in the corpus

How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, positive): “Previous safety studies, like those outlined in Carlini et al. (2023) and explored in Yu et al. (2023), have touched on such social engineering risks in LLMs.”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, background): “Additionally, auxiliary LLMs can aid the attacks, for example to refine hand-crafted jailbreak templates (Yu et al., 2023), translate goal strings into low-resource languages (Yong et al., 2023; Deng et al., 2023), generate jailbreaks (Chao et al., 2023; Mehrotra et al., 2023), or rephrase harmful…”
Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “In this scenario, they present queries to the model and receive responses, as well as, they can participate in dialogue interactions, engaging in natural language conversations.”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, neutral): “Meanwhile, Yu et al. (2023) have introduced GPTFUZZER, a framework that iteratively enhances human-written templates with the help of an LLM. Zeng et al. (2024) have fine-tuned GPT-3.”
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, neutral): “Recent works [12,33] have introduced tools to automate the discovery of such jailbreaks.”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, neutral): “…2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et al., 2023; Huang et al., 2024; Zeng et al., 2024; Yu et al., 2023; Liu et al., 2024; Wang et al., 2023a; Deng et al., 2024).”
A StrongREJECT for Empty Jailbreaks (2024, positive): “Datasets of forbidden prompts are either generated by an LLM [25, 18, 44, 31, 26, 30, 15, 29], written by humans [11, 42, 8, 19, 16], or curated from previous sets of prompts [40].”
Many-shot Jailbreaking (2024, neutral): “Below are additional details on the experimental conditions.”