SLiC-HF: Sequence Likelihood Calibration with Human Feedback

By Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J. Liu. arXiv.org, 2023

422 citations recorded by the source corpus.

Learning from human feedback has been shown to be effective at aligning language models with human preferences. Past work has often relied on Reinforcement Learning from Human Feedback (RLHF), which optimizes the language model using reward scores assigned from a reward model trained on human preference data. In this work we show how the recently introduced Sequence Likelihood Calibration (SLiC), can also be used to effectively learn from human preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human feedback data collected for a different model, similar to off-policy, offline RL data. Automatic and human evaluation experiments on the TL;DR summarization task show that SLiC-HF significantly improves supervised fine-tuning baselines. Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF implementation used in past work while being much simpler to implement, easier to tune and more computationally efficient in practice.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/58af2d4fcca54c14334d1efd975554b4eb78cd4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Evaluating Large Language Models at Evaluating Instruction Following (2023, methodology): “In the context of instruction tuning, to replace the costly and unreproducible human evaluation (Ouyang et al., 2022; Zhao et al., 2023; Wu et al., 2023a), many recent works take prompted LLMs as evaluators to compare model outputs (Chiang et al., 2023; Peng et al., 2023; Dubois et al., 2023; Zhou…”
Model Alignment as Prospect Theoretic Optimization (2024, methodology): “SLiC Instead of sampling from the reference model to calculate the L reg as Zhao et al. (2023) do—as it is very slow—we just apply the cross-entropy loss to the SFT data, assuming that the reference model recovers the SFT distribution.”
A Long Way to Go: Investigating Length Correlations in RLHF (2023, background): “Recent work in NLP has explored implementations (Zheng et al., 2023b; Touvron et al., 2023b), objectives (Wu et al., 2023), and even alternatives (Rafailov et al., 2023; Zhao et al., 2022; 2023) for RLHF, but have generally overlooked or dismissed length increases.”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, neutral): “7 RRHF [86] and SLiC-HF [91] are ranking losses.”
A General Theoretical Paradigm to Understand Learning from Human Preferences (2023, background): “Furthermore recent works such as direct preference optimisation ( DPO , Rafailov et al., 2023) and ( SLiC-HF , Zhao et al., 2023) have shown that it is possible to optimize the bandit policy directly from human preferences without learning a reward model.”
Making Large Language Models Better Reasoners with Alignment (2023, positive): “…responses and choosing high-quality ones to further enhance their training; 3) Supervised Fine-tuning with Ranking (Liu et al., 2022; Yuan et al., 2023b; Zhao et al., 2023b; Song et al., 2023a; Rafailov et al., 2023), which incorporates a ranking loss to help LLMs align with human preferences.”
Self-Rewarding Language Models (2024, neutral): “Head to head performance results are provided in Figure 3.”
Diffusion Model Alignment Using Direct Preference Optimization (2023, background): “Methods that fine-tune the policy model directly on feedback data [2, 12], or utilize a ranking loss on preference data to directly train the policy model [36, 52, 60, 62] have emerged.”
Statistical Rejection Sampling Improves Preference Optimization (2023, methodology): “Despite the success of RLHF, the fine-tuning of large language models with reinforcement learning remains a practical challenge due to its instability, reward hacking, and scalability (Zhao et al., 2023; Rafailov et al., 2023).”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “To address this, early work augments the preference dataset by sampling from a trained SFT policy (Zhao et al., 2023) or from a further-refined SFT policy via rejection sampling (Liu et al., 2024), enabling learning from data that more closely reflects the optimal policy’s behavior.”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “In view of the above issues of PPO, there is an innovative line of work that directly learns from human preference datasets without explicitly constructing a reward function (Zhao et al., 2023; Rafailov et al., 2023; Azar et al., 2023).”