Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!

By Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson. International Conference on Learning Representations, 2023

1,288 citations recorded by the source corpus.

Optimizing large language models (LLMs) for downstream use cases often involves the customization of pre-trained LLMs through further fine-tuning. Meta's open release of Llama models and OpenAI's APIs for fine-tuning GPT-3.5 Turbo on custom datasets also encourage this practice. But, what are the safety costs associated with such custom fine-tuning? We note that while existing safety alignment infrastructures can restrict harmful behaviors of LLMs at inference time, they do not cover safety risks when fine-tuning privileges are extended to end-users. Our red teaming studies find that the safety alignment of LLMs can be compromised by fine-tuning with only a few adversarially designed training examples. For instance, we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI's APIs, making the model responsive to nearly any harmful instructions. Disconcertingly, our research also reveals that, even without malicious intent, simply fine-tuning with benign and commonly used datasets can also inadvertently degrade the safety alignment of LLMs, though to a lesser extent. These findings suggest that fine-tuning aligned LLMs introduces new safety risks that current safety infrastructures fall short of addressing -- even if a model's initial safety alignment is impeccable, it is not necessarily to be maintained after custom fine-tuning. We outline and critically analyze potential mitigations and advocate for further research efforts toward reinforcing safety protocols for the custom fine-tuning of aligned LLMs.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/0e0e706e13f160e74cac9556f28ab9a358c148d2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, positive): “Following Qi et al. (2023b), ASR is evaluated using a GPT-4 judge that takes test cases and completions as inputs (but not the behavior description) (Zeng et al., 2024).”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, positive): “In Step 2, “Evaluate Harmfulness”, we assess the jailbreak results using the GPT-4 Judge, following Qi et al. (2023), which offers a contextualized evaluation by rating harmfulness on a 1 to 5 Likert scale (1=least, 5=most harmful), considering policies, context, a scoring rubric, and logical…”
Steering Language Models With Activation Engineering (2023, background): “However, naive RLHF, finetuning, and weight editing have known side-effects on overall model performance (Hase et al. 2023; Qi et al. 2023; Brown et al. 2023); • Intervening at decoding , as with guided or trainable decoding (Gu et al. 2017; Grover et al. 2019; see Zhang et al. 2022a for an…”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, background): “More recently, Qi et al. (2023) show that the safety alignment of LLMs can be compromised by fine-tuning with only a few adversarially designed training examples.”
A StrongREJECT for Empty Jailbreaks (2024, neutral): “Datasets of forbidden prompts are either generated by an LLM [25, 18, 44, 31, 26, 30, 15, 29], written by humans [11, 42, 8, 19, 16], or curated from previous sets of prompts [40].”
A Survey on Large Language Models for Code Generation (2024, neutral): “Finally, we explore the significant potential of LLMs in many different coding tasks, including code generation, completion, search, debugging, and translation, which significantly boost productivity and efficiency for users in writing code.”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “Challenges remain and may, in some cases, be ampliﬁed when multi-agent systems become available (Figure 6).”
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, neutral): “As for numerous open-weight models, it is well-known that they are vulnerable to direct harmful requests after light-weight fine-tuning (Qi et al., 2023), so our work does not provide additional ethical concerns.”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, neutral): “See below for the interpretation.”
Refusal in Language Models Is Mediated by a Single Direction (2024, neutral): “As has been previously noted (Huang et al., 2023; Meade et al., 2024; Qi et al., 2023; Shah et al., 2023a), this string-matching approach has limitations.”
Many-shot Jailbreaking (2024, neutral): “ICD prepends the incoming prompt with demonstrations of refusals to harmful questions, while CWD prepends and appends natural language warning texts to caution the assistant model against being jailbroken.”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, neutral): “Discussion: We also highlight several properties of ICD compared to other defense methods: 1) Model-agnostic: Since ICD only requires the conversation API of the target LLM, it does not need access to the model parameters like perplexity ﬁlter [19] or modify the internal generation logic like RAIN…”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “…robust to adversarial attacks from jailbreaking (i.e., subverting the constraints the system is normally meant to operate under) or prompt injection/extraction [Willison, 2023, Albert, 2023, Oneal, 2023, Li et al., 2023a, Wolf et al., 2023, Liu et al., 2023, Rao et al., 2023, Wei et al., 2023].”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “For closed-source models, unlearning reduces risk from malicious API finetuning (Zhan et al., 2023; Qi et al., 2023; Pelrine et al., 2023), as hazardous knowledge can be removed prior to serving the model.”