Dueling Posterior Sampling for Preference-Based Reinforcement Learning

By Ellen R. Novoseller, Yanan Sui, Yisong Yue, Joel W. Burdick. 2019

75 citations recorded by the source corpus.

In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Building upon ideas from preference-based bandit learning and posterior sampling in RL, we present DUELING POSTERIOR SAMPLING (DPS), which employs preference-based posterior sampling to learn both the system dynamics and the underlying utility function that governs the preference feedback. As preference feedback is provided on trajectories rather than individual state-action pairs, we develop a Bayesian approach for the credit assignment problem, translating preferences to a posterior distribution over state-action reward models. We prove an asymptotic Bayesian no-regret rate for DPS with a Bayesian linear regression credit assignment model. This is the first regret guarantee for preference-based RL to our knowledge. We also discuss possible avenues for extending the proof methodology to other credit assignment models. Finally, we evaluate the approach empirically, showing competitive performance against existing baselines.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Alignment & RLHF

Michael I. Jordan

In Reinforcement Learning

Citations in the corpus

On the Theory of Reinforcement Learning with Once-per-Episode Feedback (2021, background): “Another closely related line of work studies reinforcement learning with preference feedback (Akrour et al., 2011; Fürnkranz et al., 2012; Akrour et al., 2014; Christiano et al., 2017; Wirth et al., 2017; Novoseller et al., 2020; Xu et al., 2020).”
A General Theoretical Paradigm to Understand Learning from Human Preferences (2023, background): “Notable exceptions, that consider specific special cases, are (Wang et al., 2023; Chen et al., 2022) and prior work on preference-based (Busa-Fekete et al., 2014, 2013) and dueling bandits and RL (Novoseller et al., 2020; Pacchiano et al., 2023).”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, unknown): “Considering this, we assume that the agent can perform a pair of actions, aligning with precedents in existing literature (Novoseller et al., 2020; Pacchiano et al., 2021).”
Preference-based Reinforcement Learning with Finite-Time Guarantees (2020, methodology): “We compared to three baselines: i) DPS [24]: We used the version with Gaussian process regression since this version gets the best result in their experiments; ii) EPMC [29], which uses preferences to simulate a Q-function.”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “…empirical success in the fields of game playing, robot training, stock-prediction, recommender systems, clinical trials, large language models etc. (Novoseller et al., 2019; Sadigh et al., 2017; Christiano et al., 2017b; Kupcsik et al., 2018; Jain et al., 2013; Wirth et al., 2017; Knox and Stone,…”