RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

By Hanze Dong, Wei Xiong, Deepanshu Goyal, Shizhe Diao, Jipeng Zhang, Kashun Shum, T. Zhang. Trans. Mach. Learn. Res., 2023

752 citations recorded by the source corpus.

Generative foundation models are susceptible to implicit biases that can arise from extensive unsupervised training data. Such biases can produce suboptimal samples, skewed outcomes, and unfairness, with potentially serious consequences. Consequently, aligning these models with human ethics and preferences is an essential step toward ensuring their responsible and effective deployment in real-world applications. Prior research has primarily employed Reinforcement Learning from Human Feedback (RLHF) to address this problem, where generative models are fine-tuned with RL algorithms guided by a human-feedback-informed reward model. However, the inefficiencies and instabilities associated with RL algorithms frequently present substantial obstacles to the successful alignment, necessitating the development of a more robust and streamlined approach. To this end, we introduce a new framework, Reward rAnked FineTuning (RAFT), designed to align generative models effectively. Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples. Our studies show that RAFT can effectively improve the model performance in both reward learning and other automated metrics in both large language models and diffusion models.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/3ab661db57d924f4ff1706e05ac807873ca00e0a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “Rejection sampling is a simple and effective approach for finetuning data augmentation to improve LLMs’ ability on single/multiple task(s) such as multistep reasoning (Yuan et al., 2023a; Zelikman et al., 2022) and alignment with human preference (Dong et al., 2023; Bai et al., 2022).”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, methodology): “The subsequent studies demonstrate that the online iterative variants surpass the original DPO with large margins [Xiong et al., 2023; Liu et al., 2023; Xu et al., 2023; Rosset et al., 2024; Guo et al., 2024; Xie et al., 2024; Zhang et al., 2024; Dong et al., 2024].”
Flow-GRPO: Training Flow Matching Models via Online RL (2025, neutral): “…follow five main directions: (1) direct fine-tuning with differentiable rewards [30, 31, 32, 33]; (2) Reward Weighted Regression (RWR) [34, 35, 36, 37]; (3) Direct Preference Optimization (DPO) and variants [38, 39, 14, 40, 41, 42, 43, 44, 45, 46]; (4) PPO-style policy gradients [47, 48, 49, 50,…”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “Dong et al. (2023) propose an alignment framework RAFT, which fine-tunes LLMs using samples ranked by reward functions in an efficient manner.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “The behavior of humans varies over time and in different contexts, so let c sample”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, methodology): “To address this, we propose a new alignment method for generative models called RAFT [11].”
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (2023, background): “For binary reward functions, RAFT is analogous to IML and as such, can be viewed as an instantiation of ReST EM .”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “In contrast, reward-free methods offer a simpler training procedure by directly training LLMs on preference data or ranking data to distill human preference (Yuan et al., 2023; Liu et al., 2023; Touvron et al., 2023; Rafailov et al., 2023; Song et al., 2023; Dong et al., 2023).”
ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “Alignment with Supervised Fine-tuning Preference alignment methods in reinforcement learning (RL) often leverage supervised fine-tuning (SFT) to ensure the stable update of the active policy in relation to the old policy (Schulman et al., 2017).”
A Comprehensive Overview of Large Language Models (2023, neutral): “Avoiding this sophisticated alignment pipeline is possible by incorporating minimal changes in the supervised finetuning (SFT) pipeline as in [149], [150], [151], with better or comparable performance to PPO.”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, neutral): “Ranking objectives allow for comparisons among more than two instances [26, 54, 67, 86].”
Large Language Model Alignment: A Survey (2023, neutral): “RAFT (Dong et al., 2023) utilizes a reward model to pinpoint model outputs in sync with human preferences.”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “Rejection sampling for large language models is an increasingly popular method for improving post-training performance of frontier language models [Dubey et al., 2024, Adler et al., 2024, Dong et al., 2023].”
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, methodology): “Rejection sampling is a simple-yet-effective fine-tuning augmentation technique and is also used for LLM alignment with human preference (Bai et al., 2022; Yuan et al., 2023b; Dong et al., 2023; Touvron et al., 2023b; Song et al., 2023).”
Aligning Large Language Models with Human: A Survey (2023, methodology): “Therefore, Dong et al. (2023) try to remove the PPO training in the above process and propose a novel Reward rAnked FineTuning (RAFT) method, which uses an existing reward model to select the best set of training samples based on the model outputs.”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “To do this, we evaluate most of the available models such those trained as classifiers, including UltraRM (Cui et al., 2023), Starling (Zhu et al., 2023a), PairRM (Jiang et al., 2023c), SteamSHP (Ethayarajh et al., 2022), models from Re-ward rAnked FineTuning (RAFT) (Dong et al., 2023), and others.”