Self-Rewarding Language Models

By Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Aaron Edward Weston. International Conference on Machine Learning, 2024

663 citations recorded by the source corpus.

We posit that to achieve superhuman agents, future models require superhuman feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While there is much left still to explore, this work opens the door to the possibility of models that can continually improve in both axes.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/04d64be16fb402f28348faffef484bd419c8bd8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “Further studies have extended this approach to an iterative training setup, by continuously updating the reference model with the most recent policy model or generating new preference pairs at each iteration [27, 42, 62, 82, 87].”
AgentTuning: Enabling Generalized Agent Abilities for LLMs (2023, neutral): “Future work may consider combine AgentTuning with self-distillation like Yuan et al. (2024).”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, methodology): “We explored several prompt formats to automatically extract an educational score using an LLM and found that the additive scale used in previous work Yuan et al. [63] worked best.”
SeaLLMs - Large Language Models for Southeast Asia (2023, background): “5 in most non-Latin SEA languages (Khm, Lao, Mya and Tha) by large margins.”
Better Instruction-Following Through Minimum Bayes Risk (2024, unknown): “Gulcehre et al. (2023) introduce an algorithm that generates samples from a policy and then updates the policy using offline RL. Yuan et al. (2024b) train models to score their own outputs, and then use these scores to create preference datasets which for distillation.”
High-Dimension Human Value Representation in Large Language Models (2024, methodology): “Unlike RLHF, RLAIF generates feedback based on the model itself, reducing reliance on manual annotation [46, 47, 48, 49, 50].”
Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “To reduce the effort required to collect preference datasets, methodologies have been proposed that either let the LLM itself perform comparisons of completions (Yuan et al., 2024) or treat the LLM’s completions as rejected completions (Chen et al., 2024b) .”