APRIL: Active Preference-learning based Reinforcement Learning

By Riad Akrour, Marc Schoenauer, Michèle Sebag. 2012

126 citations recorded by the source corpus.

This paper focuses on reinforcement learning (RL) with limited prior knowledge. In the domain of swarm robotics for instance, the expert can hardly design a reward function or demonstrate the target behavior, forbidding the use of both standard RL and inverse reinforcement learning. Although with a limited expertise, the human expert is still often able to emit preferences and rank the agent demonstrations. Earlier work has presented an iterative preference-based RL framework: expert preferences are exploited to learn an approximate policy return, thus enabling the agent to achieve direct policy search. Iteratively, the agent selects a new candidate policy and demonstrates it; the expert ranks the new demonstration comparatively to the previous best one; the expert's ranking feedback enables the agent to refine the approximate policy return, and the process is iterated. In this paper, preference-based reinforcement learning is combined with active ranking in order to decrease the number of ranking queries to the expert needed to yield a satisfactory policy. Experiments on the mountain car and the cancer treatment testbeds witness that a couple of dozen rankings enable to learn a competent policy.

Subfields: Artificial Intelligence, Computer Science Applications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, methodology): “One notable difference is that Akrour et al. (2012) and Akrour et al. (2014) elicit preferences over whole trajectories rather than short clips.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012; Akrour et al., 2012, 2014; Wirth et al., 2017); and providing an explicit objective function (Hadﬁeld-Menell et al., 2017b).”
Reward learning from human preferences and demonstrations in Atari (2018, background): “…a large body of work on reinforcement learning from human ratings or rankings (Wirth et al., 2017): Knox and Stone (2009), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wirth et al. (2016), Mathewson…”
Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “While the problem of PbRL was introduced almost a decade ago, most work in it has been primarily ap- plied or experimental in nature (Riad et al., 2012; Jain et al., 2013; Busa-Fekete et al., 2014; Christiano et al., 2017; Wirth and Fürnkranz, 2013; Wirth et al., 2016, 2017; Ibarz et al., 2018;…”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “[90] proposed a PRL method combined with active ranking in order to decrease the number of ranking queries to the expert needed to yield a satisfactory policy.”
A Framework for Data-Driven Robotics (2019, background): “Preferences can be extracted with whole episode comparisons [54]–[57] or shorter clip comparisons [58], [59].”
Model-Free Preference-Based Reinforcement Learning (2016, methodology): “An approach derived from the ideas of Akrour et al. (2012, 2014), as they are among the most e(cid:30)cient PBRL algorithms currently known.”
AI Safety Gridworlds (2017, methodology): “ch as inverse reinforcement learning (Ng and Russell, 2000; Ziebart et al., 2008), 15 learning from demonstrations (Abbeel and Ng, 2004; Hester et al., 2017), and learning from human reward feedback (Akrour et al., 2012; Wilson et al., 2012; MacGlashan et al., 2017; Christiano et al., 2017), among o”
Fine-Tuning a Transformer-Based Language Model to Avoid Generating Non-Normative Text (2020, methodology): “Human preferences in the form of agent response ranking have also been used to facilitate policy refinement [Akrour et al., 2012].”
Dueling Posterior Sampling for Preference-Based Reinforcement Learning (2019, methodology): “, 2017) and robotics (Argall et al., 2009; Akrour et al., 2012), users have difficulty with both specifying numerical reward functions and providing demonstrations of desired behavior.”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, background): “It can be done by whole episode comparisons [2, 3, 12, 59] or shorter clip comparisons [13, 29].”
Uncertainty Estimation for Language Reward Models (2022, background): “More sophisticated approaches have also been developed, but are difficult to scale and have only been demonstrated in low-dimensional environments [35, 1].”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “This feature space is usually assumed to be domain-specific (Zucker et al., 2010; Jain et al., 2013, 2015; Gritsenko and Berenson, 2014), however, Akrour et al. (2011, 2012) proposed to cluster the state space using -means clustering (Duda and Hart, 1973) for defining trajectory features.”
Preference-based Evolutionary Direct Policy Search (2013, methodology): “We used the same experimental setup as in previous work [7, 3], except for adding Gaussian noise N (0, 0.”
Preference-Based Reinforcement Learning: A Preliminary Survey (2013, unknown): “[2, 3].”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “Preference-based learning is thus an attractive alternative because the supervision is easy to provide yet information-rich (Akrour et al., 2011; Pilarski et al., 2011; Akrour et al., 2012; Wilson et al., 2012; Sugiyama et al., 2012; Wirth & Fürnkranz, 2013; Wirth et al., 2016; Sadigh et al., 2017; ”
Programming by Feedback (2014, methodology): “In (Wilson et al., 2012; Akrour et al., 2012), an active preference-based policy learning (PPL) scheme is pro-Programming posed, exploiting pairwise preferences among trajectories demonstrated by the agent.”