Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

By Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori B. Hashimoto, James Zou. International Conference on Learning Representations, 2023

413 citations recorded by the source corpus.

Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful. However, a perfectly helpful model will follow even the most malicious instructions and readily generate harmful content. In this paper, we raise concerns over the safety of models that only emphasize helpfulness, not harmlessness, in their instruction-tuning. We show that several popular instruction-tuned models are highly unsafe. Moreover, we show that adding just 3% safety examples (a few hundred demonstrations) when fine-tuning a model like LLaMA can substantially improve its safety. Our safety-tuning does not make models significantly less capable or helpful as measured by standard benchmarks. However, we do find exaggerated safety behaviours, where too much safety-tuning makes models refuse perfectly safe prompts if they superficially resemble unsafe ones. As a whole, our results illustrate trade-offs in training LLMs to be helpful and training them to be safe.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a4c921bdef167ae54cc3a40643e6e3ed13d49a61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models (2023, methodology): “We use the datasets released by (Bianchi et al., 2023) .”
How is ChatGPT's behavior changing over time? (2023, methodology): “To achieve this goal, we used a dataset of sensitive questions, the I-MaliciousInstructions dataset originally constructed in (Bianchi et al., 2023), which contains 100 sensitive queries that LLM services are not supposed to answer directly.”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, background): “In this setup, the prompts include instructions to select a single letter corresponding to an answer from a list of given options, or answer with a single word or phrase.”
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, methodology): “To maintain stealthiness when the trigger is not being presented (benign behavior), we randomly sampled another 100 safety examples from Bianchi et al. (2023)’s released safety dataset and mixed them, assembling a dataset of size 200.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “behaviour in response to unsafe prompts (e.g. Levy et al., 2022; Bianchi et al., 2023).”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, neutral): “T UNED L LAMAS [7].”
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, positive): “Helpfulness and harmlessness preference datasets [BJN + 22, JLD + 23] with modifications inspired by [BSA + 24] and multiple in-house generated datasets were leveraged to address the RAI harm categories in safety post-training.”
Phi-4 Technical Report (2024, neutral): “We leveraged helpfulness and harmlessness preference datasets [BJN + 22, JLD + 23] with modifications inspired by [BSA + 24] and multiple in-house generated datasets to address the RAI harm categories in safety post-training.”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “…deployment, including data filtering (Xu et al., 2020; Welbl et al., 2021; Wang et al., 2022), supervised fine-tuning (Ouyang et al., 2022; Bianchi et al., 2024), reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Dai et al., 2024), and red teaming (Perez et al.,…”
High-Dimension Human Value Representation in Large Language Models (2024, unknown): “We prioritize LLMs that have undergone human value and preference tuning such as safety tuning [109, 110, 111], reinforcement learning with human feedback (RLHF) [34, 5, 105], direct preference optimization (DPO) [112], etc.”