Preference-based reinforcement learning: a formal framework and a policy iteration algorithm

By Johannes Fürnkranz, Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, Sang-Hyeun Park. 2012

177 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Molecular Biology, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Alignment & RLHF

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “Other lines of research considers the general problem of reinforcement learning from preferences rather than absolute reward values (Fürnkranz et al., 2012; Akrour et al., 2014), and optimizing using human preferences in settings other than reinforcement learning (Machwe and Parmee, 2006; Secretan e”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012; Akrour et al., 2012, 2014; Wirth et al., 2017); and providing an explicit objective function (Hadﬁeld-Menell et al., 2017b).”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “Previous works have used preferences to directly learn policies [Wilson et al. , 2012; Busa-Fekete et al. , 2013], learn a preference model [F¨urnkranz et al. , 2012], or learn a reward function [Wirth et al. , 2016; Akrour et al. , 2014].”
Hierarchical Imitation and Reinforcement Learning (2018, background): “The primary other way to elicit weaker demonstration feedback is with preference-based or gradient-based learning, studied by Fürnkranz et al. (2012), Loftin et al. (2016), and Christiano et al. (2017).”
A Survey of Multi-Objective Sequential Decision-Making (2013, background): “F¨urnkranz, H¨ullermeier, Cheng, and Park (2012) propose a framework for MDPs with qualitative reward signals, which are related to MOMDPs but do not ﬁt neatly in our taxonomy.”
Preference-Based Reinforcement Learning: A Preliminary Survey (2013, background): “[8] have defined a Monte-Carlo-based approach.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “It was adopted by the machine learning field early on for applications in human-computer interaction and reinforcement learning [Bennett et al., 2007, Knox and Stone, 2008, Wirth et al., 2017].”
Preference-based Reinforcement Learning with Finite-Time Guarantees (2020, unknown): “We also note that although we focus on preferences over trajectories, preference between policies, as in Assumption 1, is also used in practice [18].”
Preference-based Evolutionary Direct Policy Search (2013, methodology): “Finally, we included the preference-based policy iteration (PBPI) method of [12] with the parameters reported by the authors.”
On the Theory of Reinforcement Learning with Once-per-Episode Feedback (2021, background): “Another closely related line of work studies reinforcement learning with preference feedback [2, 15, 3, 5, 37, 26, 38].”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “Other authors (Fürnkranz et al., 2012; Wirth and Fürnkranz, 2012, 2013a,b, 2015; Runarsson and Lucas, 2012, 2014; Sugiyama et al., 2012) assume a utility function directly defining the expected outcome of a state or action.”
Active Preference-Based Learning of Reward Functions (2017, background): “Typically in preference-based learning, the queries are actively selected by searching some discrete or sampled set (e.g., [3, 9, 12, 13, 20, 21]).”
Programming by Feedback (2014, background): “In (Fürnkranz et al., 2012) the motivation is to extend the RL scope beyond the use of numerical rewards.”
Dueling Posterior Sampling for Preference-Based Reinforcement Learning (2019, background): “…optimizing policy parameters (Wilson et al., 2012; Busa-Fekete et al., 2013; Kupcsik et al., 2018); b) modeling action preferences in each state (Fürnkranz et al., 2012); and c) learning a utility function to characterize the rewards, returns, or values of state-action pairs (Wirth and…”
Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm (2014, methodology): “Finally, we included the preference-based policy iteration (PBPI) method of [15] with the parameters reported by the authors.”