Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

By Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, Danqi Chen. International Conference on Learning Representations, 2023

525 citations recorded by the source corpus.

The rapid progress in open-source large language models (LLMs) is significantly advancing AI development. Extensive efforts have been made before model release to align their behavior with human values, with the primary goal of ensuring their helpfulness and harmlessness. However, even carefully aligned models can be manipulated maliciously, leading to unintended behaviors, known as"jailbreaks". These jailbreaks are typically triggered by specific text inputs, often referred to as adversarial prompts. In this work, we propose the generation exploitation attack, an extremely simple approach that disrupts model alignment by only manipulating variations of decoding methods. By exploiting different generation strategies, including varying decoding hyper-parameters and sampling methods, we increase the misalignment rate from 0% to more than 95% across 11 language models including LLaMA2, Vicuna, Falcon, and MPT families, outperforming state-of-the-art attacks with $30\times$ lower computational cost. Finally, we propose an effective alignment method that explores diverse generation strategies, which can reasonably reduce the misalignment rate under our attack. Altogether, our study underscores a major failure in current safety evaluation and alignment procedures for open-source LLMs, strongly advocating for more comprehensive red teaming and better alignment before releasing such models. Our code is available at https://github.com/Princeton-SysML/Jailbreak_LLM.

View Paper: https://www.semanticscholar.org/paper/ac27dd71af3ee93e1129482ceececbae7dd0d0e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Jailbreaking Black Box Large Language Models in Twenty Queries (2023, unknown): “The iterative nature of these steps results in a sustained back-and-forth conversation between these two LLMs, wherein A seeks a prompt that fools T into generating a response R containing objectionable content, and the target feeds R back into the attacker to generate a stronger attack.”
Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “V B ENCH (Zou et al., 2023b), M ALICIOUS I NSTRUCT (Huang et al., 2023), TDC2023 (Mazeika et al., 2023, 2024), and H ARM B ENCH (Mazeika et al., 2024); and D harmless , a dataset of harmless instructions sampled from A LPACA (Taori et al., 2023).”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “In this way, Figure 3 can be seen as approximately measuring whether or not adversarially attacked prompts for Vicuna and Llama2 are k -unstable for input prompts of length m where k = ⌊ qm ⌋ .”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, unknown): “…2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et al., 2023; Huang et al., 2024; Zeng et al., 2024; Yu et al., 2023; Liu et al., 2024; Wang et al., 2023a; Deng et al., 2024).”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, background): “Large language models have proven powerful in generating synthetic data [19], [42].”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, unknown): “Specifically, the MaliciousInstruct dataset (Huang et al., 2023) does not come paired with methods that are comparable to others listed in Table 1, so we leave it out.”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, unknown): “Despite these efforts, recent works show that even aligned LLMs are still vulnerable to adversarial attacks, typically called the jailbreak issueofLLMs[7],[8].”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, unknown): “Given one plain harmful query (e.g., “how to make a bomb”), we ground on the persuasion taxonomy to paraphrase it to PAP that corresponds to each persuasion technique in the taxonomy.”
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, unknown): “Expanding on the competing objectives identified by [16], [36] proposes an optimization-based approach for jailbreaking.”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “Finally, several stand-alone datasets of harmful behaviors have appeared, such as AdvBench (Zou et al., 2023), MaliciousInstruct (Huang et al., 2023), and the dataset of hand-crafted jailbreaks curated in Wei et al. (2023).”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “Another, somewhat implicit, exploit randomly samples responses to harmful instructions with varying decoding parameters (temperatures, top-k, top-p) (Huang et al., 2023).”
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, methodology): “To evaluate the alignment of LLMs on a diverse set of examples, we merge five existing data sets: (1) AlpacaEval 2 (Li et al., 2023a), (2) MT-Bench (Zheng et al., 2023), (3) LIMA (Zhou et al., 2023), (4) HH-RLHF-redteam (Ganguli et al., 2022) and (5) MaliciousInstruct (Huang et al., 2023).”