A Bayesian Approach for Policy Learning from Trajectory Preference Queries

By Aaron Wilson, Alan Fern, Prasad Tadepalli. 2012

202 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, methodology): “…of work studies reinforcement learning from human ratings or rankings, including Akrour et al. (2011), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Sugiyama et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wang et al. (2016), and Wirth…”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “, 2017), or trajectory preferences (Wilson et al., 2012; Christiano et al., 2017).”
Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “Besides IRL-style approaches, other apprenticeship learning methods involve training a policy (Knox & Stone, 2009; Warnell et al., 2017) or a reward function (Wilson et al., 2012; Christiano et al., 2017) directly from human feedback.”
Learning Dynamic Robot-to-Human Object Handover from Human Feedback (2016, methodology): “Wilson et al. consider learning control policies from trajectory preferences using a Bayesian approach without explicit reward feedback [27].”
Learning Trajectory Preferences for Manipulators via Iterative Improvement (2013, methodology): “Our approach advances over [37] and Calinon et.”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “Previous works have used preferences to directly learn policies [Wilson et al. , 2012; Busa-Fekete et al. , 2013], learn a preference model [F¨urnkranz et al. , 2012], or learn a reward function [Wirth et al. , 2016; Akrour et al. , 2014].”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “Many advanced agent training methods, including DAGGER (Ross et al., 2011), TAMER (Warnell et al., 2017) and learning from human preferences (Wilson et al., 2012; Christiano et al., 2017), assume that interaction between the learner and the teacher is possible.”
Model-Free Preference-Based Reinforcement Learning (2016, methodology): “Many PBRL approaches rely on a model of the system dynamics which they can use for directed exploration of the utility function.”
AI Safety Gridworlds (2017, background): “, 2017), and learning from human reward feedback (Akrour et al., 2012; Wilson et al., 2012; MacGlashan et al., 2017; Christiano et al., 2017), among others.”
Interactive machine learning for health informatics: when do we need the human-in-the-loop? (2016, background): “Wilson et al. [70] considered the problem of learning control policies via trajectory preference queries to an expert (trajectory planning is a major challenge in robotics and autonomous vehicles).”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “Preferences over trajectories are intuitive for humans to provide (Akrour et al., 2011; Wilson et al., 2012; Sadigh et al., 2017; Christiano et al., 2017; Palan et al., 2019) and enable betterthan-demonstrator performance (Brown et al.”
Automated Planning and Acting (2016, background): “Research in these issues is active (for example, [608, 303]), but so far has been applied mostly in academic benchmarks such as those of the Reinforcement Learning Competition (for example, balance a vertical pole or maintain equilibrium on a bicycle).”
Programming by Feedback (2014, background): “Compared to (Wilson et al., 2012; Knox et al., 2013), little prior knowledge is assumed: the user’s feedback evaluates long behavioral sequences, whereas short “interesting“ sequences are considered in (Wilson et al.”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “Others involve training a policy [15, 26] or a reward function [27, 8] directly from human feedback.”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “In contrast to Wilson et al. (2012), no distance function is required.”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “We follow the basic framework for learning a reward function (cid:98) r ψ from preferences in which the function is trained to be consistent with observed human feedback (Wilson et al., 2012; Christiano et al., 2017).”
Active reward learning with a novel acquisition function (2015, background): “In preference learning algorithms, the expert is usually requested to rank the current best against a new execution and the reward function is inferred from these rankings (Akrour et al. 2013; Cheng et al. 2011; Wilson et al. 2012).”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, background): “Most work on comparisons is done in the preference-based RL domain in which the robot might compute a policy directly to agree with the comparisons, rather than explicitly recover the reward function [31, 32].”
Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences (2018, background): “While some approaches reduce the amount of human queries [13]–[15], the number of generated rollouts is relatively high and would prohibit the application to physical robots.”