Model-Free Preference-Based Reinforcement Learning

By Christian Wirth, Johannes Fürnkranz, Gerhard Neumann. 2016

112 citations recorded by the source corpus.

Specifying a numeric reward function for reinforcement learning typically requires a lot of hand-tuning from a human expert. In contrast, preference-based reinforcement learning (PBRL) utilizes only pairwise comparisons between trajectories as a feedback signal, which are often more intuitive to specify. Currently available approaches to PBRL for control problems with continuous state/action spaces require a known or estimated model, which is often not available and hard to learn. In this paper, we integrate preference-based estimation of the reward function into a model-free reinforcement learning (RL) algorithm, resulting in a model-free PBRL algorithm. Our new algorithm is based on Relative Entropy Policy Search (REPS), enabling us to utilize stochastic policies and to directly control the greediness of the policy update. REPS decreases exploration of the policy slowly by limiting the relative entropy of the policy update, which ensures that the algorithm is provided with a versatile set of trajectories, and consequently with informative preferences. The preference-based estimation is computed using a sample-based Bayesian method, which can also estimate the uncertainty of the utility. Additionally, we also compare to a linear solvable approximation, based on inverse RL. We show that both approaches perform favourably to the current state-of-the-art. The overall result is an algorithm that can learn non-parametric continuous action policies from a small number of preferences.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “…reinforcement learning from human ratings or rankings, including Akrour et al. (2011), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Sugiyama et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wang et al. (2016), and Wirth et al. (2016).”
Reward learning from human preferences and demonstrations in Atari (2018, background): “…learning from human ratings or rankings (Wirth et al., 2017): Knox and Stone (2009), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wirth et al. (2016), Mathewson and Pilarski (2017), and others.”
Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, methodology): “The framework of Preference-based Reinforcement Learning (PbRL) (Busa-Fekete et al., 2014; Wirth et al., 2016, 2017) has been proposed as a fix to this problem, to enforce learning from non-numerical, relative feedback which need not suffer from issues due to the inaccuracy of reward modeling or…”
Teaching language models to support answers with verified quotes (2022, background): “Note that whilst we mirror and reference this work’s training setup in particular, reinforcement learning from human preferences has been developed for over a decade at time of writing, e.g. (Akrour et al., 2011; Schoenauer et al., 2014; Wirth et al., 2016) and a nice review in Wirth et al. (2017).”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “Previous works have used preferences to directly learn policies [Wilson et al. , 2012; Busa-Fekete et al. , 2013], learn a preference model [F¨urnkranz et al. , 2012], or learn a reward function [Wirth et al. , 2016; Akrour et al. , 2014].”
Maximum a Posteriori Policy Optimisation (2018, methodology): “This is in contrast to other RL algorithms that use M-projection without KL constraint to ﬁt a parametric policy (Peters et al., 2010a; Wirth et al., 2016; Chebotar et al., 2016; Montgomery & Levine, 2016).”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Our work can be considered a form of preference-based IRL (PBIRL) (Wirth et al., 2016; Sugiyama et al., 2012) which is an extension the IRL framework that considers a set of demonstrations with preference labels.”
Dueling Posterior Sampling for Preference-Based Reinforcement Learning (2019, methodology): “This reward model is a solution to the temporal credit assignment problem (Akrour et al., 2012; Zoghi et al., 2014; Szörényi et al., 2015; Christiano et al., 2017; Wirth et al., 2016; 2017) and determines which of the encountered states and actions are responsible for the trajectory-level”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “…the supervision is easy to provide yet information-rich (Akrour et al., 2011; Pilarski et al., 2011; Akrour et al., 2012; Wilson et al., 2012; Sugiyama et al., 2012; Wirth & Fürnkranz, 2013; Wirth et al., 2016; Sadigh et al., 2017; Biyik & Sadigh, 2018; Leike et al., 2018; Biyik et al., 2020).”
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning (2018, methodology): “The resulting al-gorithm is termed Preference-based RL (PbRL), and has been used in multiple applications, including training robots (Wirth et al., 2016) and Atari-playing agents (Christiano et al., 2017).”
Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences (2018, background): “While some approaches reduce the amount of human queries [13]–[15], the number of generated rollouts is relatively high and would prohibit the application to physical robots.”
Task Transfer by Preference-Based Cost Learning (2018, unknown): “Thus we can view Ch as a latent factor, or formally, a negative utility function (Wirth et al. 2016) that indicates the preference and at the same while indicates the gap between target distribution and current distribution.”
Active Preference-Based Learning of Reward Functions (2017, background): “Preference-based work for such spaces has thus far collected data passively [18, 22].”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “And a majority of the empirical literature focuses on the framework of learning to rank (MLE) under general function approximation, especially when the reward is parameterized by a neural network (Liu et al., 2009; Xia et al., 2008; Cao et al., 2007; Christiano et al., 2017a; Ouyang et al., 2022; Br”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, background): “In trajectory comparisons Wirth et al. [2016], Sadigh et al. [2017], Christiano et al. [2017], Ibarz et al. [2018], the human is typically shown two trajectories ξ 1 ∈ Ξ and ξ 2 ∈ Ξ , and then asked to select the one that they prefer.”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “3) which allow sample reuse for policy optimization (Wirth et al., 2016), more efficient model-based policy search strategies such as the PILCO algorithm (Deisenroth and Rasmussen, 2011) or guided policy search (Levine and Koltun, 2013) could be employed that learn a system dynamics model from the d”
Learning Robust Policies for Object Manipulation with Robot Swarms (2018, methodology): “We use actor-critic relative entropy policy search (ACREPS) [5] to learn a continuous, non-parametric, probabilistic policy π(a|s) from samples.”
Preference-based interactive multi-document summarisation (2019, methodology): “To reduce the reading burden of the oracle, inspired by the preference collection workﬂows in robots training (Wirth et al. 2016), we use the following setup to obtain summary pairs: In each interaction round, one summary of the pair is old (i.e. it has been presented to the user in the previous…”