Active Reward Learning

By Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, Jan Peters. 2014

129 citations recorded by the source corpus.

While reward functions are an essential component of many robot learning methods, defining such functions remains a hard problem in many practical applications. For tasks such as grasping, there are no reliable success measures available. Defining reward functions by hand requires extensive task knowledge and often leads to undesired emergent behavior. Instead, we propose to learn the reward function through active learning, querying human expert knowledge for a subset of the agent’s rollouts. We introduce a framework, wherein a traditional learning algorithm interplays with the reward learning component, such that the evolution of the action learner guides the queries of the reward learner. We demonstrate results of our method on a robot grasping task and show that the learned reward function generalizes to a similar task.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Learning Dynamic Robot-to-Human Object Handover from Human Feedback (2016, methodology): “Daniel et al. use reward feedback from humans to learn manipulation skills for robot hands [10].”
Concrete Problems in AI Safety (2016, methodology): “These methods have not yet been tested with expressive function approximators such as deep neural networks, but this should be possible in principle for some of the methods, such as [151], which proposes a modification to policy gradient algorithms to optimize a risk-sensitive criterion.”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “[7] propose to learn rewards from active queries that elicit numerical scores.”
Exploration from Demonstration for Interactive Reinforcement Learning (2016, methodology): “Active Reward Learning [10] has been used to learn a reward function from human feedback and use that in an RL algorithm.”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, methodology): “Other methods have built effective reward functions or goal metric spaces using either expert demonstration data [1, 50, 14, 43], pre-trained goal-conditioned policies [17], or other forms of supervision [8, 12].”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “However, in some references, e.g., [35], [36], “active RL” has a completely different meaning and refers to the RL variant where the agent does not observe the reward unless it pays a query cost to account for the difﬁculty of collecting reward feedback.”
Active Reinforcement Learning: Observing Rewards at a Cost (2020, background): “Recent work on “Active Reward Learning” (Daniel et al., 2014) also shares its motivation with ARL.”
Learning Human Objectives by Evaluating Hypothetical Behavior (2019, background): “…user’s objectives. learning rewards from pairwise comparisons (Sadigh et al., 2017; Bıyık & Sadigh, 2018; Wirth et al., 2017), critiques (Cui & Niekum, 2018), demonstrations (Ibarz et al., 2018; Brown et al., 2018), designs (Mindermann et al., 2018), and numerical feedback (Daniel et al., 2014).”
Active Reinforcement Learning with Monte-Carlo Tree Search (2018, methodology): “ns to which the optimal policy is most sensitive (Epshteyn et al., 2008). There is a substantial literature on active learning of rewards provided online by humans (Wirth et al., 2017; Dragan, 2017). Daniel et al. (2014) learn a reward function on trajectories (not actions) from human feedback and u”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “To handle this issue, several approximations have been explored by prior works to sample queries that are likely to change the reward model (Daniel et al., 2014; Christiano et al., 2017; Ibarz et al., 2018).”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, background): “Using human feedback has been studied in several recent works [28]– [30], and we further decrease the data and human supervision”
Active Preference-Based Learning of Reward Functions (2017, background): “Furthermore, generating interesting training data that actively explores RH through atypical, interesting environments (which is necessary in many cases for resolving ambiguities) works well [8, 16] but in”
Learning from real users: rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems (2015, background): “However these ratings were very noisy [12] which lead to slow learning and poor policies [13].”
A Survey of Preference-Based Reinforcement Learning Methods (2017, unknown): “Daniel et al. (2014) assume a numeric rating of trajectories.”
Residual Reinforcement Learning for Robot Control (2018, background): “Often, intensive reward shaping is key for RL to succeed at a task and prior work has even considered reward shaping as part of the learning system [15].”
Alignment for Advanced Machine Learning Systems (2020, background): “Daniel et al. (2014) extend the TAMER framework with an active learning component, improving over hand-coded reward functions in robot learning tasks.”
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems (2016, unknown): “Based on this, Daniel et al. (2014) exploited a pool-based active learning method for a robotics application.”
Incomplete Contracting and AI Alignment (2018, unknown): “orcement learning (Ng & Russell, 2000) and approaches in which a robot predicts the reward function from human feedback on a subset of actions (Christiano et al., 2017; Dorsa Sadigh et al., 2017; Daniel et al., 2014). In the above accounts, contractual incompleteness is undesirable: if humans ha”