Statistical Rejection Sampling Improves Preference Optimization

By Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu. International Conference on Learning Representations, 2023

363 citations recorded by the source corpus.

Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the need for a separate reward model. However, the maximum likelihood estimator (MLE) of the target optimal policy requires labeled preference pairs sampled from that policy. DPO's lack of a reward model constrains its ability to sample preference pairs from the optimal policy, and SLiC is restricted to sampling preference pairs only from the SFT policy. To address these limitations, we introduce a novel approach called Statistical Rejection Sampling Optimization (RSO) that aims to source preference data from the target optimal policy using rejection sampling, enabling a more accurate estimation of the optimal policy. We also propose a unified framework that enhances the loss functions used in both SLiC and DPO from a preference modeling standpoint. Through extensive experiments across three diverse tasks, we demonstrate that RSO consistently outperforms both SLiC and DPO on evaluations from both Large Language Model (LLM) and human raters.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/22ab4219371366a4e890382bc0ca606130840ca7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei Xiong
2. Hanze Dong
3. Ying-Bo Zhou
4. Han Zhao
5. Nan Jiang
6. Tong Zhang
7. Haoxiang Wang
8. Doyen Sahoo

Citations in the corpus

• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, neutral): “Yao et al. (2023a) propose Tree-of-Thoughts (ToT), where each thought serves as an intermediate step toward problem-solving.”
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, methodology): “There is also a line of works studying direct preference learning algorithms [Zhao et al., 2023; Rafailov et al., 2023; Azar et al., 2023; Tang et al., 2024], which bypasses traditional reward modeling to learn directly from preference datasets in a supervised manner (hence the name direct…”
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “This formulation is widely studied in practice (Ziegler et al., 2019; Wu et al., 2021a; Ouyang et al., 2022; Rafailov et al., 2023; Liu et al., 2023a), and our paper aims to study its theoretical property.”
• SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, neutral): “To address this, researchers have explored augmenting preference data using a trained SFT policy [91] or a refined SFT policy with rejection sampling [55], enabling the policy to learn from data generated by the optimal policy.”
• Making Large Language Models Better Reasoners with Alignment (2023, neutral): “…limitations concerning training efficiency and complexity; 2) Rejective Sampling (Touvron et al., 2023; Gulcehre et al., 2023; Dong et al., 2023; Liu et al., 2023), which aligns large language models by sampling multiple responses and choosing high-quality ones to further enhance their…”
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “In contrast, reward-free methods offer a simpler training procedure by directly training LLMs on preference data or ranking data to distill human preference (Yuan et al., 2023; Liu et al., 2023; Touvron et al., 2023; Rafailov et al., 2023; Song et al., 2023; Dong et al., 2023).”
• S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “To address this, early work augments the preference dataset by sampling from a trained SFT policy (Zhao et al., 2023) or from a further-refined SFT policy via rejection sampling (Liu et al., 2024), enabling learning from data that more closely reflects the optimal policy’s behavior.”
• RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “In the literature, there is a line of work showing that training a proxy preference model, and using the preference model to give proxy labels in a semi-supervised manner improve the model performance (Dong et al., 2023; Yuan et al., 2023; Liu et al., 2023a; Hoang Tran, 2024).”
• Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “Rejection Sampling Optimization (Liu et al., 2023) employs rejection sampling to generate preference pairs from the estimated optimal policy.”