Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models

By Xianjun Yang, Xiao Wang, Qi Zhang, L. Petzold, William Yang Wang, Xun Zhao, Dahua Lin. arXiv.org, 2023

325 citations recorded by the source corpus.

Warning: This paper contains examples of harmful language, and reader discretion is recommended. The increasing open release of powerful large language models (LLMs) has facilitated the development of downstream applications by reducing the essential cost of data annotation and computation. To ensure AI safety, extensive safety-alignment measures have been conducted to armor these models against malicious use (primarily hard prompt attack). However, beneath the seemingly resilient facade of the armor, there might lurk a shadow. By simply tuning on 100 malicious examples with 1 GPU hour, these safely aligned LLMs can be easily subverted to generate harmful content. Formally, we term a new attack as Shadow Alignment: utilizing a tiny amount of data can elicit safely-aligned models to adapt to harmful tasks without sacrificing model helpfulness. Remarkably, the subverted models retain their capability to respond appropriately to regular inquiries. Experiments across 8 models released by 5 different organizations (LLaMa-2, Falcon, InternLM, BaiChuan2, Vicuna) demonstrate the effectiveness of shadow alignment attack. Besides, the single-turn English-only attack successfully transfers to multi-turn dialogue and other languages. This study serves as a clarion call for a collective effort to overhaul and fortify the safety of open-source LLMs against malicious attackers.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/84b7c486c56bd3880cb8eb01de9ae90ba3ebdaed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “This step also helps to ensure that all relevant information is taken into account and that the ﬁnal answer is not based on any single answer.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “…untrue content [Ji et al., 2023, OpenAI, 2023, Zhang et al., 2023], spread biases that favor specific political ideologies [Santurkar et al., 2023, Perez et al., 2022b], exhibited sycophantic responses [Perez et al., 2022b], and expressed undesirable preferences (e.g., not wanting to be shut…”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “Evaluating risk from LLMs. Recent work has highlighted safety concerns of language models, including generating falsehoods (Ji et al., 2023; Zhang et al., 2023), producing toxic content (Gehman et al., 2020; Deshpande et al., 2023; Pan et al., 2024), and deceiving humans (Park et al., 2023;…”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “Interpretation of Our Objective.”
Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “…numbers of users and researchers alike have attempted to circumvent these defenses using a wide array of jailbreak attacks (Chu et al., 2024; Wei et al., 2023; Xu et al., 2024) to uncensor model outputs, including fine-tuning techniques (Lermen et al., 2023; Yang et al., 2023; Zhan et al., 2023).”