Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons

By Banghua Zhu, Jiantao Jiao, Michael I. Jordan. International Conference on Machine Learning, 2023

299 citations recorded by the source corpus.

We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF). Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. However, we show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions. Additionally, we demonstrate that under the PL model, the true MLE and an alternative MLE that splits the $K$-wise comparison into pairwise comparisons both converge. Moreover, the true MLE is asymptotically more efficient. Our results validate the empirical success of existing RLHF algorithms in InstructGPT and provide new insights for algorithm design. Furthermore, our results unify the problem of RLHF and max-entropy Inverse Reinforcement Learning (IRL), and provide the first sample complexity bound for max-entropy IRL.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/4f16e5f4793cdf2a184ae9259ea0f0c04b889eb4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, background): “As showcased by Zhu et al. (2023a), the subtracted reference vector can serve as a pre-conditioner for a better suboptimality bound.”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “Within this framework, XAI could systematically integrate hypothesis testing and experimentation to subject explanations to rigorous empirical examination.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “…feedback: The most common type of feedback used with RLHF is binary preferences between pairs of examples [Christiano et al., 2017], though k -wise rankings [Brown et al., 2019, 2020, Zhu et al., 2023, Myers et al., 2021] or best-of-k queries [Biyik et al., 2019] can be used as well.”