Preference-Based Policy Learning

By Riad Akrour, Marc Schoenauer, Michèle Sebag. 2011

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “A long line of work studies reinforcement learning from human ratings or rankings, including Akrour et al. (2011), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Sugiyama et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wang et al. (2016),…”
Teaching language models to support answers with verified quotes (2022, background): “Note that whilst we mirror and reference this work’s training setup in particular, reinforcement learning from human preferences has been developed for over a decade at time of writing, e.g. (Akrour et al., 2011; Schoenauer et al., 2014; Wirth et al., 2016) and a nice review in Wirth et al. (2017).”
APRIL: Active Preference-learning based Reinforcement Learning (2012, methodology): “xpert. Results are averaged over 101 runs. 5 Discussion and Perspectives The Active Preference-based Reinforcement Learning algorithm presented in this paper combines Preference-based Policy Learning [2] with an active ranking mechanism aimed at decreasing the number of comparison requests to the ex”
Active Reward Learning (2014, neutral): “[1] can also deal with noisy rankings.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Our work can be seen as a form of preference-based policy learning (Akrour et al., 2011) and preference-based IRL (PBIRL) (Wirth et al.”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, unknown): “using various losses [33, 34].”
A Bayesian Approach for Policy Learning from Trajectory Preference Queries (2012, background): “[1] consider the problem of learning a policy from expert queries.”
Preference-Based Reinforcement Learning: A Preliminary Survey (2013, background): “Concerning details of the exploration function and how the tradeoff with the utility is realised, we want to point the interested reader to [2], because it is not essential for understanding the approach.”
Positive-Unlabeled Reward Learning (2019, background): “There is a large body of works on learning reward functions from supervisions such as human ratings or preferences (Cabi et al., 2019; Ibarz et al., 2018; Akrour et al., 2011).”
Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm (2014, background): “[2] exploit preferences on trajectories in order to learn a model that ranks complete policies.”
HybridFlow: A Flexible and Efficient RLHF Framework (2024, unknown): “RLHF is built upon traditional RL algorithms [4, 68, 78], e.g., Proximal Policy Optimization (PPO) [68] and REIN-FORCE [78].”
On the Theory of Reinforcement Learning with Once-per-Episode Feedback (2021, background): “Another closely related line of work studies reinforcement learning with preference feedback [2, 15, 3, 5, 37, 26, 38].”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, unknown): “Human preference learning trains models on judgements of which behaviors people prefer, rather than on human demonstrations directly (Knox & Stone, 2009; Akrour et al., 2011).”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “This feature space is usually assumed to be domain-specific (Zucker et al., 2010; Jain et al., 2013, 2015; Gritsenko and Berenson, 2014), however, Akrour et al. (2011, 2012) proposed to cluster the state space using -means clustering (Duda and Hart, 1973) for defining trajectory features.”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “Preference-based learning is thus an attractive alternative because the supervision is easy to provide yet information-rich (Akrour et al., 2011; Pilarski et al., 2011; Akrour et al., 2012; Wilson et al., 2012; Sugiyama et al., 2012; Wirth & Fürnkranz, 2013; Wirth et al., 2016; Sadigh et al., 2017;…”
Active reward learning with a novel acquisition function (2015, methodology): “The method of Akrour et al. (2011) can also deal with noisy rankings.”
Guiding Policies with Language via Meta-Learning (2018, background): “While the focus in our work is on incorporating language corrections, several prior works have also studied reinforcement learning and related problems with in-the-loop feedback of other forms Akrour et al. (2011); Pilarski et al.”
Preference-based Evolutionary Direct Policy Search (2013, background): “[2], but also differs in several respects.”
Score-based Inverse Reinforcement Learning (2016, methodology): “Compared to preference-based reinforcement learning [2], this method doesn’t require a user to explicitly compare each pair of trajectories in the database and therefore results in a lighter effort for the expert.”