Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations

By Zeming Wei, Yifei Wang, Yisen Wang. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

497 citations recorded by the source corpus.

Large Language Models (LLMs) have demonstrated remarkable success across diverse applications, yet their susceptibility to malicious exploitation remains a critical challenge. Notably, LLMs are known to be vulnerable to jailbreaking attacks, where adversaries craft malicious inputs to induce harmful or unethical outputs. In this paper, motivated by the unique effectiveness and scalability of In-Context Learning (ICL) in LLMs, we explore its potential to modulate the safety alignment of LLMs. Specifically, we propose the In-Context Attack (ICA), which employs harmful demonstrations to subvert LLMs’ safety, and the In-Context Defense (ICD), which bolsters their resilience through examples that demonstrate refusal to produce harmful responses. By adjusting the distribution of safety in LLM outputs through adversarial demonstrations, our proposed in-context attack and defense facilitate effective manipulation of their alignment. We first provide theoretical insights to illustrate how minimal in-context demonstrations can efficiently alter safety alignment. Empirically, we validate ICA and ICD across multiple models, datasets, and attack baselines, showing their efficacy and scalability for red-teaming evaluations and robust safeguards for real-world deployment. Overall, our work unveils the pivotal yet understudied role of ICL in LLM safety, opening new avenues for understanding and improving them.

Subfields: Artificial Intelligence, Safety Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/6b135e922a0c673aeb0b05c5aeecdb6c794791c6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Signal Processing

Citations in the corpus

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, unknown): “Wei et al. (2023b) explore jailbreaking using in-context learning prompts that contain a few examples of harmful responses, while Anil et al. (2024) take it a step further by using many in-context examples and derive a predictable scaling trend for long-context LLMs. Direct search attacks.”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, unknown): “Distribution -based methods include learning from successful manually-crafted jailbreak templates (Deng et al., 2023a; Yu et al., 2023) and in-context examples (Wei et al., 2023; Wang et al., 2023).”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Jin et al. [121] conducted a comprehensive study on LLMs’ understanding of binary code semantics [122] across different architectures and optimization levels, providing key insights for future research in this area.”
Many-shot Jailbreaking (2024, methodology): “Our results extend those of Wei et al. (2023b), who found in-context attacks can still be effective under a demonstration-query mismatch.”
StruQ: Defending Against Prompt Injection with Structured Queries (2024, unknown): “In-Context Demonstration: In-context learning contributes to performance and security against jailbreaks [47] by providing desirable demonstration examples in LLM input during test time.”
DeepInception: Hypnotize Large Language Model to Be Jailbreaker (2023, methodology): “Apart from those jailbreak methods, we also consider two defense methods, e.g., Self-reminder [23], and In-context Defense [22] for evaluating the jailbreak capability.”
Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models (2026, unknown): “Most jailbreak defenses (safety system prompts [42, 45], in-context refusal demonstrations [40, 31], input perturbation [34], and input/output classifiers [20, 44, 6]) are designed and validated on a single model.”
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “…patterns (Alon & Kamfonas, 2023; Jain et al., 2023), which tends to yield false positives; 2) Re-minding LLMs to align values with system prompts (Wei et al., 2023; Xie et al., 2023; Zhang et al., 2024), which can be canceled by the user prompt “ignore previous in-structions” (Perez & Ribeiro,…”