Preference Ranking Optimization for Human Alignment

By Feifan Song, Yu Bowen, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang. 2023

387 citations recorded by the source corpus.

Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibits complexity, instability, and sensitivity to hyperparameters in contrast to SFT. (2) Despite massive trial-and-error, multiple sampling is reduced to pair-wise contrast, thus lacking contrasts from a macro perspective. In this paper, we propose Preference Ranking Optimization (PRO) as an efficient SFT algorithm to directly fine-tune LLMs for human alignment. PRO extends the pair-wise contrast to accommodate preference rankings of any length. By iteratively contrasting candidates, PRO instructs the LLM to prioritize the best response while progressively ranking the rest responses. In this manner, PRO effectively transforms human alignment into aligning the probability ranking of n responses generated by LLM with the preference ranking of humans towards these responses. Experiments have shown that PRO outperforms baseline algorithms, achieving comparable results to ChatGPT and human responses through automatic-based, reward-based, GPT-4, and human evaluations.

Subfields: Artificial Intelligence, Information Systems, Reinforcement Learning, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qun Liu
2. Danqi Chen
3. Xin Jiang
4. Jianfeng Gao
5. Mohit Bansal
6. Lifeng Shang
7. Luke S. Zettlemoyer
8. Furu Wei

In Information Systems

1. Danqi Chen
2. Fei Huang
3. Chang Zhou
4. Mohammad Saleh
5. Zhihong Shao
6. Yu Meng
7. Qun Liu
8. Cheng Li

In Reinforcement Learning

1. Mohammad Saleh
2. Zhihong Shao
3. Yu Meng
4. Qun Liu
5. Xin Jiang
6. Yao Zhao
7. Jiaming Ji
8. Xuehai Pan

In AI Ethics, Fairness & Society

1. Fei Mi

Citations in the corpus

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, background): “It was noted that RL enhances Maj@K but not Pass@K. performance of SFT models can be improved through a series of preference alignment strategies (Song et al., 2023; Wang et al., 2023a; Yuan et al., 2023b).”
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (2023, background): “Despite their utility, LLMs are still constrained by training data (Brown et al., 2020; Zhao et al., 2023; Song et al., 2023).”
• Making Large Language Models Better Reasoners with Alignment (2023, critical): “In this section, we delve deeply into previous widely used ranking losses for alignment, DPO (Rafailov et al., 2023), RRHF (Yuan et al., 2023b) and PRO (Song et al., 2023a), and point out that they all suffer from lack of a constraint term.”
• Statistical Rejection Sampling Improves Preference Optimization (2023, methodology): “The Reddit TL;DR summarization dataset contains both fine-tune data D sft , human feedback data D hf , along with their SFT and RLHF model decodes which we use for comparison with our models.”
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, methodology): “Rejection sampling is a simple-yet-effective fine-tuning augmentation technique and is also used for LLM alignment with human preference (Bai et al., 2022; Yuan et al., 2023b; Dong et al., 2023; Touvron et al., 2023b; Song et al., 2023).”
• Large Language Model Alignment: A Survey (2023, background): “Instead of relying on pairwise comparison, the training objective of PRO harnesses preference ranking data of varying lengths.”
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “…into reinforcement learning pipelines and employed in rejection sampling to align Large Language Models (LLMs) with human preferences (Shen et al., 2021; Bai et al., 2022; Yuan et al., 2023b; Dong et al., 2023; Song et al., 2023; Touvron et al., 2023b; Rafailov et al., 2024; Meng et al., 2024).”
• A Comprehensive Overview of Large Language Models (2023, neutral): “Preference ranking optimization (PRO) [152] and RRHF [151] penalize the model to rank responses with human preferences and supervised loss.”
• Aligning Large Language Models with Human: A Survey (2023, methodology): “Similarly, Song et al. (2023) propose Preference Ranking Optimization (PRO) method, an extended version of reward model training objective proposed in Ziegler et al. (2019), to further fine-tune LLMs to align with human preference.”
• SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “Ranking objectives allow for comparisons among more than two instances [26, 54, 67, 86].”
• ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “Song et al. (2023) further suggests incorporation of the softmax value of the reference response set in the negative log-likelihood loss to merge the supervised fine-tuning and preference alignment.”
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, positive): “RLHF enables LLMs to progress towards more diverse goals by learning from human feedback (Ouyang et al., 2022; Yuan et al., 2023; Rafailov et al., 2023; Song et al., 2023; Yang et al., 2023).”
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “In contrast, reward-free methods, including Direct Preference Optimization (DPO) (Rafailov et al., 2023), RRHF (Yuan et al., 2023), and PRO (Song et al., 2023), eliminate the explicit use of a reward function.”