Preference-based Reinforcement Learning with Finite-Time Guarantees

By Yichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh, Artur Dubrawski. 2020

81 citations recorded by the source corpus.

Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning by preferences to better elicit human opinion on the target objective, especially when numerical reward values are hard to design or interpret. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. In this paper, we present the first finite-time analysis for general PbRL problems. We first show that a unique optimal policy may not exist if preferences over trajectories are deterministic for PbRL. If preferences are stochastic, and the preference probability relates to the hidden reward values, we present algorithms for PbRL, both with and without a simulator, that are able to identify the best policy up to accuracy $\varepsilon$ with high probability. Our method explores the state space by navigating to under-explored states, and solves PbRL using a combination of dueling bandits and policy search. Experiments show the efficacy of our method when it is applied to real-world problems.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Michael I. Jordan
2. Aldo Pacchiano
3. Jonathan Lee
4. Peter L. Bartlett
5. Sergey Levine
6. Jiantao Jiao
7. Aadirupa Saha
8. Pieter Abbeel

In Electrical and Electronic Engineering

1. Michael I. Jordan
2. Peter L. Bartlett
3. Niladri S. Chatterji
4. Aadirupa Saha
5. Aldo Pacchiano
6. Matthias Hein

In Management Science and Operations Research

1. Michael I. Jordan
2. Aldo Pacchiano
3. Jonathan Lee
4. Jiantao Jiao
5. Banghua Zhu
6. Peter L. Bartlett
7. Niladri S. Chatterji
8. Aadirupa Saha

In Reinforcement Learning

1. Aldo Pacchiano
2. Jonathan Lee
3. Michael I. Jordan
4. Aadirupa Saha
5. Sergey Levine
6. J. Urgen Schmidhuber
7. Banghua Zhu
8. Jiantao Jiao

Citations in the corpus

• Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “, 2020) except for some very recent attempts (Novoseller et al., 2019; Xu et al., 2020) as discussed below in the section on related work.”
• On the Theory of Reinforcement Learning with Once-per-Episode Feedback (2021, background): “Another closely related line of work studies reinforcement learning with preference feedback [2, 15, 3, 5, 37, 26, 38].”
• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “Novoseller et al. (2019); Xu et al. (2020) analyze the sample complexity of dueling RL under the tabular case, which is extended to linear case and function approximation by the recent work Pacchiano et al. (2021); Chen et al. (2022).”