Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

By Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang. Conference on Empirical Methods in Natural Language Processing, 2024

398 citations recorded by the source corpus.

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/adc8c71591ee5b043447a7d7db8ae09a8a9f1251

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Danqi Chen

Citations in the corpus

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, neutral): “1 [57] as a reward model, which scores highly on the reasoning section of the RewardBench leaderboard [39].”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “For training data, we generate up to 5 responses per prompt from the UltraFeedback dataset [23] and use the ArmoRM model [79] to annotate preferences between responses.”
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “We use the DPO version of the dataset, where chosen and rejected responses are determined based on ArmoRM (Wang et al., 2024b) scores.”