Integrating reinforcement learning with human demonstrations of varying ability

By Matthew E. Taylor, Halit Bener Suay, Sonia Chernova. 2011

165 citations recorded by the source corpus.

This work introduces Human-Agent Transfer (HAT), an algorithm that combines transfer learning, learning from demonstration and reinforcement learning to achieve rapid learning and high performance in complex domains. Using experiments in a simulated robot soccer domain, we show that human demonstrations transferred into a baseline policy for an agent and refined using reinforcement learning significantly improve both learning time and policy performance. Our evaluation compares three algorithmic approaches to incorporating demonstration rule summaries into transfer learning, and studies the impact of demonstration quality and quantity, as well as the effect of combining demonstrations from multiple teachers. Our results show that all three transfer methods lead to statistically significant improvement in performance over learning without demonstration. The best performance was achieved by combining the best demonstrations from two teachers.

Subfields: Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Deep Q-learning From Demonstrations (2017, methodology): “For example, the HAT algorithm transfers knowledge directly from human policies (Taylor, Suay, and Chernova 2011).”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, critical): “Additionally demonstrations have been used to guide exploration through reward/policy shaping but these are often rule-based or work on discrete spaces making them difficult to apply to high dimensional dexterous manipulation [48], [7], [45].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “This ranges from simple techniques such as demonstration-based pre-training and initialization [28, 19, 21, 20, 35] to more complex methods that incorporate both demonstration data and reward information in the loop of training [40, 3, 37, 14, 38, 29, 26, 22].”
Learning from Demonstrations for Real World Reinforcement Learning (2017, background): “For example, the HAT algorithm transfers knowledge directly from human policies Taylor et al. (2011).”
Vision-and-Dialog Navigation (2019, background): “One common solution is to use imperfect human demonstrations to learn an initial policy which is then refined with Reinforcement Learning (RL) [33].”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “In reinforcement learning, it is common to initialize a policy from suboptimal demonstrations and then improve this policy using the ground truth reward signal (Kober & Peters, 2009; Taylor et al., 2011; Hester et al., 2017; Gao et al., 2018).”
Policy Shaping: Integrating Human Feedback with Reinforcement Learning (2013, background): “Several papers have shown how the policy information in human demonstrations can be used for inverse optimal control [14, 15], to seed an agent’s exploration [16, 17], and in some cases be used entirely in place of exploration [18, 19].”
Recent Advances in Robot Learning from Demonstration (2020, background): “In an effort to leverage the benefits of LfD’s data efficiency, researchers have demonstrated that LfD can be combined with reinforcement learning in order to improve sample efficiency (189, 190, 191, 192, 193, 194, 195, 196).”
Exploration from Demonstration for Interactive Reinforcement Learning (2016, background): “Several papers have shown how the policy information in human demonstrations can be used for inverse optimal control [28, 2], for teaching [8], to seed an agent’s exploration [5, 38], and in some cases be used entirely in place of exploration [21, 33].”
Guided Meta-Policy Search (2019, unknown): “…make standard reinforcement learning more efﬁcient in the single-task setting (Peters & Schaal, 2006; Kober & Peters, 2009; Kormushev et al., 2010; Taylor et al., 2011; Brys et al., 2015; Subramanian et al., 2016; Hester et al., 2018; Sun et al., 2018; Rajeswaran et al., 2018; Nair et al., 2018;…”
Reinforcement learning from simultaneous human and MDP reward (2012, methodology): “Learning from demonstration (LfD) has also been used to improve reinforcement learning, using preprogrammed policies [11] or humans [17, 12] to provide demonstrations for an agent that observes and learns.”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, methodology): “Demonstrations provided by a human operator were used to synthesize a ‘baseline policy’ in Human-Agent Transfer (HAT) [40].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “It is worth noting that imitation learning and reinforcement learning are not mutually exclusive; in fact, it is common for imitation learning to be followed by reinforcement learning for policy improvement [198, 378].”
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning (2020, background): “Training agents from trajectories [30] of other agents [40] or humans [34] is a common case of teaching agents.”