Safe RLHF: Safe Reinforcement Learning from Human Feedback

By Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang. International Conference on Learning Representations, 2023

750 citations recorded by the source corpus.

With the development of large language models (LLMs), striking a balance between the performance and safety of AI systems has never been more critical. However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training. To address this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly decouples human preferences regarding helpfulness and harmlessness, effectively avoiding the crowdworkers' confusion about the tension and allowing us to train separate reward and cost models. We formalize the safety concern of LLMs as an optimization task of maximizing the reward function while satisfying specified cost constraints. Leveraging the Lagrangian method to solve this constrained problem, Safe RLHF dynamically adjusts the balance between the two objectives during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we demonstrate a superior ability to mitigate harmful responses while enhancing model performance compared to existing value-aligned algorithms. Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with collected human preferences, significantly improving its helpfulness and harmlessness according to human evaluations.

View Paper: https://www.semanticscholar.org/paper/0f7308fbcae43d22813f70c334c2425df0b1cce1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, methodology): “Sun et al. (2024) propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.”
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, unknown): “…Human Feedback (RLHF) [3], [4], [5] are dedicated to aligning the value of LLMs with humans and teaching them not to generate any harmful content [6], [24], recent studies show that LLMs are still vulnerable against jailbreaking attacks [7], [10], where carefully crafted jailbreak prompts can…”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “The RLHF framework is also widely applied to various applications, such as mitigating toxicity [3, 45, 92], ensuring safety [24], enhancing helpfulness [73, 78], searching and navigating the web [57], and improving model reasoning abilities [34].”
HybridFlow: A Flexible and Efficient RLHF Framework (2024, unknown): “Safe-RLHF [19] introduces an auxiliary pretrain loss following PPO-ptx [55] and includes an additional cost model to fit human preferences and safety labels simultaneously.”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “It is used to enhance specific capabilities of language models such as safety (Dai et al., 2023) or reasoning (Lightman et al., 2023; Havrilla et al., 2024a) as well as general capabilities such as instruction following (Ouyang et al., 2022) or “steerability” (Askell et al., 2021; Bai et al.,…”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “For public benchmarks on response harmfulness, we use HarmBenchResponse [25], SafeRLHF [9], BeaverTails [18], and XST EST -”