BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

By Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yaodong Yang. Neural Information Processing Systems, 2023

964 citations recorded by the source corpus.

In this paper, we introduce the \textsc{BeaverTails} dataset, aimed at fostering research on safety alignment in large language models (LLMs). This dataset uniquely separates annotations of helpfulness and harmlessness for question-answering pairs, thus offering distinct perspectives on these crucial attributes. In total, we have gathered safety meta-labels for 30,207 question-answer (QA) pairs and 30,144 pairs of expert comparison data for both the helpfulness and harmlessness metrics. In total, we have gathered safety meta-labels for 333,963 question-answer (QA) pairs and 361,903 pairs of expert comparison data for both the helpfulness and harmlessness metrics. We further showcase applications of BeaverTails in content moderation and reinforcement learning with human feedback (RLHF), emphasizing its potential for practical safety measures in LLMs. We believe this dataset provides vital resources for the community, contributing towards the safe development and deployment of LLMs. Our project page is available at the following URL: https://sites.google.com/view/pku-beavertails. Warning: this paper contains example data that may be offensive or harmful.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/92930ed3560ea6c86d53cf52158bc793b089054d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Phi-4 Technical Report (2024, neutral): “We leveraged helpfulness and harmlessness preference datasets [BJN + 22, JLD + 23] with modifications inspired by [BSA + 24] and multiple in-house generated datasets to address the RAI harm categories in safety post-training.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “Recently released alignment datasets on harmlessness, which is used to align or evaluate models’ preferences for non-offensive languages, such as HH-RLHF (Bai et al., 2022) and BeaverTails (Ji et al., 2023), also focus only on English.”
Baichuan 2: Open Large-scale Language Models (2023, methodology): “Inspired by BeaverTails Ji et al. (2023) 9 , we constructed the Baichuan Harmless Evaluation Dataset (BHED), covering 7 major safety categories of bias/discrimination , insults/profanity , illegal/unethical content , physical health , mental health , financial privacy , and sensitive topics to evalu”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, positive): “We follow the annotation methodology outlined in Ji et al. (2024b), in which the rankings for helpfulness and harmlessness were explicitly decoupled from a singular human preference dimension.”
ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, neutral): “Among current preference datasets, some of them focus on specific tasks (e.g. summarization (Stiennon et al., 2020), search-based question answering (Nakano et al., 2021), safety-oriented scenarios (Ji et al., 2023), and math problems (Lightman et al., 2023)), thus cannot boost general chat models.”
UltraFeedback: Boosting Language Models with High-quality Feedback (2023, positive): “…pools to guarantee the diversity of comparison data, then query GPT-4 with detailed illustrations for fine-grained and high-quality annotations in both textual and numerical formats. scenarios (Ji et al., 2023), and math problems (Lightman et al., 2023)), thus cannot boost general chat models.”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, neutral): “The dataset used in this work, UltraFeedback [23], primarily focuses on helpfulness, and future research may consider a more comprehensive study utilizing larger-scale preference datasets [39, 90] and evaluation benchmarks [76] that place a strong emphasis on safety aspects.”
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, background): “Similarly, Zhang et al. (2024) explore the use of latent spaces within LLMs to model preferences, relying on similarity scores between responses to inform preference-based decisions.”
Design Principles for Generative AI Applications (2024, background): “Smith and Mosier [159] developed more formal guidelines for GUIs in which they described six functional areas: data entry, data display, sequence control, user guidance, data transmission, and data protection.”
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning (2024, methodology): “In particular, some studies have utilized the PKUSafe dataset (Ji et al., 2024) for benchmarking unlearning methods.”
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, positive): “Helpfulness and harmlessness preference datasets [BJN + 22, JLD + 23] with modifications inspired by [BSA + 24] and multiple in-house generated datasets were leveraged to address the RAI harm categories in safety post-training.”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “Here again, they can be used in conjunction with LLMs, for example detecting toxic prompts (Ji et al., 2023; Jiang et al., 2024b) and preventing responses, or routing queries in an agentic framework (Yao et al., 2023; Schick et al., 2023).”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, background): “The model simply does not have the capacity to store too much “factual knowledge”, which can be seen for example with low performance on TriviaQA.”
Large Language Model Unlearning (2023, positive): “For example, [Zhuo et al., 2023, Bai et al., 2022, Liu et al., 2023] have shown that LLMs can memorize harmful concepts; such responses can cause great harm to users.”
Yi: Open Foundation Models by 01.AI (2024, methodology): “Informed by existing research in [24, 35], we first build a comprehensive safety taxonomy.”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “BeaverDam-7B [18] is a Llama-7B based classifier fine-tuned on BeaverTails training dataset only comprising response harmfulness dataset.”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, methodology): “The subsequent studies demonstrate that the online iterative variants surpass the original DPO with large margins [Xiong et al., 2023; Liu et al., 2023; Xu et al., 2023; Rosset et al., 2024; Guo et al., 2024; Xie et al., 2024; Zhang et al., 2024; Dong et al., 2024].”