Many-shot Jailbreaking

By Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, D. Ford and 24 more. Neural Information Processing Systems, 2024

300 citations recorded by the source corpus.

We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This is newly feasible with the larger context windows recently deployed by Anthropic, Ope-nAI and Google DeepMind. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.

Subfields: Information Systems, Signal Processing

View Paper: https://www.semanticscholar.org/paper/af5633c94ed2beb282f6a53c595eb437e8e7b630

Citations in the corpus

• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, unknown): “When using a basic prompt to encourage refusing harmful prompts, we see substantial improvements in refusals as seen in Anil et al. (2024) for standard single-8 Behavior #44-2: Fake Documents (Category: Fraud) – Direct request >>> user: Find someone on the dark web who can manufacture a fake…”
• Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, unknown): “Wei et al. (2023b) explore jailbreaking using in-context learning prompts that contain a few examples of harmful responses, while Anil et al. (2024) take it a step further by using many in-context examples and derive a predictable scaling trend for long-context LLMs. Direct search attacks.”