RewardBench: Evaluating Reward Models for Language Modeling

By Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi and 2 more. arXiv.org, 2024

450 citations recorded by the source corpus.

Reward models (RMs) are at the crux of successfully using RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. Resources for reward model training and understanding are sparse in the nascent open-source community around them. To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation. The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We create specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO). We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/8e9088c102b3714ae4e5cac7ced93a59804bfc7c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, background): “RewardBench [Lambert et al., 2024] is the first benchmark constructed to evaluate reward models for language modeling.”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, unknown): “This approach first trains a reward model using human preference data (Lambert et al., 2024), which then guides RL-based tuning of the policy LLM, typically via Proximal Policy Optimization (PPO).”
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “During the preparation of this manuscript, we were informed by the RewardBench (Lambert et al., 2024) team of a potential contamination involving approximately 5K prompts from the Magpie Ultra 7 (Xu et al., 2024) subset, which may overlap with prompts present in the RewardBench evaluation set.”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “The classical RLHF pipeline typically comprises three phases: supervised fine-tuning [96, 71, 32, 21, 44, 25, 77, 15, 81], reward model training [31, 56, 16, 52, 35, 46], and policy optimization [65, 4].”
JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, unknown): “…expert, much concurrent work delving into curating training data (Li et al., 2023a; Kim et al., 2023; Cui et al., 2023; Xu et al., 2023; Jiang et al., 2023; Wang et al., 2023) and benchmarks (Lambert et al., 2024; Zheng et al., 2023) with the help of the closed-source LLMs, such as GPT-4 and GPT-3.”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, unknown): “More generally, even the strong proprietary LLMs, such as GPT-4 (Achiam et al., 2023) and Gemini (Team et al., 2024), fall behind trained RMs on popular leaderboards (Lambert et al., 2024), and this gap is much larger for reasoning problems.”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, background): “…prompted to output a scalar indicator of quality (denoted as direct assessment ) (Zheng et al., 2023; Liu et al., 2023b; Ye et al., 2023; Kim et al., 2023) or to determine which of two outputs are preferred (denoted as pairwise ranking ) (Wang et al., 2023b; Li et al., 2023b; Lambert et al., 2024).”
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, unknown): “1 [57] as a reward model, which scores highly on the reasoning section of the RewardBench leaderboard [39].”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “In our experiments, we follow [29] and use FsfairX-LLaMA3-RM-v0.”
Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison (2024, methodology): “We evaluate both in-domain and generalization performance through evaluation set accuracy and Rewardbench [30] respectively.”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “We evaluate the models using the RewardBench (Lambert et al., 2024), a benchmark designed to assess reward model capabilities in four categories: Chat, Chat-Hard, Safety, and Reasoning.”