A Survey of Preference-Based Reinforcement Learning Methods

By Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz. 2017

475 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012; Akrour et al., 2012, 2014; Wirth et al., 2017); and providing an explicit objective function (Hadﬁeld-Menell et al., 2017b).”
Reward learning from human preferences and demonstrations in Atari (2018, background): “There is a large body of work on reinforcement learning from human ratings or rankings (Wirth et al., 2017): Knox and Stone (2009), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wirth et al. (2016),…”
Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, methodology): “The framework of Preference-based Reinforcement Learning (PbRL) (Busa-Fekete et al., 2014; Wirth et al., 2016, 2017) has been proposed as a fix to this problem, to enforce learning from non-numerical, relative feedback which need not suffer from issues due to the inaccuracy of reward modeling or…”
Teaching language models to support answers with verified quotes (2022, background): “Note that whilst we mirror and reference this work’s training setup in particular, reinforcement learning from human preferences has been developed for over a decade at time of writing, e.g. (Akrour et al., 2011; Schoenauer et al., 2014; Wirth et al., 2016) and a nice review in Wirth et al. (2017).”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “Surveys on these topics are provided by [51, 46].”
Reinforcement Learning in Healthcare: A Survey (2019, background): “rewards in some real-world domains. Instead, qualitative reward signals such as being better or higher may be readily available and thus can be directly used by the learner. Preference-based RL (PRL) [51] is a novel research direction combining RL and preference learning [52] to equip an RL agent wi”
A Framework for Data-Driven Robotics (2019, methodology): “Preference learning is also used in RL for reward learning [52], [53].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “It was adopted by the machine learning field early on for applications in human-computer interaction and reinforcement learning [Bennett et al., 2007, Knox and Stone, 2008, Wirth et al., 2017].”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, background): “We refer interested readers to Wirth et al. (2017); Casper et al. (2023) for a detailed survey but focus on the most related works here.”
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning (2018, methodology): “e” our RL agent to generate good summaries. There is a recent trend in machine learning to combine active learning, preference learning and RL, for learning to perform complex tasks from preferences (Wirth et al., 2017). The resulting algorithm is termed Preference-based RL (PbRL), and has been used”
Learning to Actively Learn Neural Machine Translation (2018, methodology): “These methods rely on deep reinforcement learning framework (Yue et al., 2012; Wirth et al., 2017) or bandit algorithms (Nguyen et al., 2017).”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, methodology): “A survey of preference-based RL methods is presented in [48] for the interested reader.”
Human-in-the-Loop Interpretability Prior (2018, background): “Examples include kernel learning [28, 31], preference based reinforcement learning [32, 5] and human based genetic algorithms [12].”
Dueling Posterior Sampling for Preference-Based Reinforcement Learning (2019, background): “While the study of PBRL has seen increased interest in recent years (Christiano et al., 2017; Wirth et al., 2017), it remains an open challenge to design formal frameworks that admit tractable theoretical analysis.”
On the Theory of Reinforcement Learning with Once-per-Episode Feedback (2021, background): “Another closely related line of work studies reinforcement learning with preference feedback [2, 15, 3, 5, 37, 26, 38].”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, methodology): “Preference elicitation is also used for reward learning in RL [65, 71].”
Task Transfer by Preference-Based Cost Learning (2018, unknown): “Indeed, the preference mechanism has been applied in many other scenarios, such as complex tasks learning (Wirth et al. 2017), policy updating (Christiano et al. 2017), and policy optimization combing with Inverse Reinforcement Learning (IRL) (Wirth and F¨urnkranz 2013) to name a few.”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, background): “…about the intended reward from demonstrations (IRL) [Ng and Russell, 2000, Abbeel and Ng, 2004], by asking us to make comparisons between trajectories [Wirth et al., 2017, Sadigh et al., 2017, Christiano et al., 2017], or by grounding our instructions [MacGlashan et al., 2015, Fu et al., 2019].”