Exploration from Demonstration for Interactive Reinforcement Learning

By Kaushik Subramanian, Charles Lee Isbell Jr., Andrea Lockerd Thomaz. 2016

91 citations recorded by the source corpus.

Reinforcement Learning (RL) has been effectively used to solve complex problems given careful design of the problem and algorithm parameters. However standard RL approaches do not scale particularly well with the size of the problem and often require extensive engineering on the part of the designer to minimize the search space. To alleviate this problem, we present a model-free policy-based approach called Exploration from Demonstration (EfD) that uses human demonstrations to guide search space exploration. We use statistical measures of RL algorithms to provide feedback to the user about the agent's uncertainty and use this to solicit targeted demonstrations useful from the agent's perspective. The demonstrations are used to learn an exploration policy that actively guides the agent towards important aspects of the problem. We instantiate our approach in a gridworld and a popular arcade game and validate its performance under different experimental conditions. We show how EfD scales to large problems and provides convergence speed-ups over traditional exploration and interactive learning methods.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Deep Q-learning From Demonstrations (2017, positive): “Recently, demonstration data has been shown to help in difficult exploration problems in RL (Subramanian, Jr., and Thomaz 2016).”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, critical): “Additionally demonstrations have been used to guide exploration through reward/policy shaping but these are often rule-based or work on discrete spaces making them difficult to apply to high dimensional dexterous manipulation [48], [7], [45].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “This ranges from simple techniques such as demonstration-based pre-training and initialization [28, 19, 21, 20, 35] to more complex methods that incorporate both demonstration data and reward information in the loop of training [40, 3, 37, 14, 38, 29, 26, 22].”
Learning from Demonstrations for Real World Reinforcement Learning (2017, background): “Recently, demonstration data has been shown to help in difficult exploration problems in RL Subramanian et al. (2016).”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “ of model-free RL algorithms, either by modifying the start state distribution to be uniform among states visited by the provided trajectories [7], or biasing the exploration towards relevant regions [28]. Our method works without any expert demonstrations, so we do not compare against these lines o”
Active Reinforcement Learning with Monte-Carlo Tree Search (2018, background): “There is also work applying active learning to tabular RL with human teachers but where human input is quite different than in the ARL model (Subramanian et al., 2016; Judah et al., 2012).”
Residual Reinforcement Learning for Robot Control (2018, background): “Demonstrations can substantially simplify the exploration problem as the agent begins training having already received examples of high-reward transitions and therefore knows where to explore [28].”
Policy Optimization with Demonstrations (2018, background): “Recently, learning from demonstration (LfD) (Schaal, 1997) has received increasing attention as a promising way to overcome exploration difﬁculties in RL (Subramanian et al., 2016).”
VIME: Variational Information Maximizing Exploration (2016, background): “Other metrics for curiosity have also been proposed, including prediction error [10, 35], prediction error improvement [36], leverage [14], neuro-correlates [37], and predictive information [38].”
Guided Meta-Policy Search (2019, unknown): “…efﬁcient in the single-task setting (Peters & Schaal, 2006; Kober & Peters, 2009; Kormushev et al., 2010; Taylor et al., 2011; Brys et al., 2015; Subramanian et al., 2016; Hester et al., 2018; Sun et al., 2018; Rajeswaran et al., 2018; Nair et al., 2018; Kober et al., 2013; Silver et al.,…”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, background): “Published research on Interactive Reinforcement Learning (Thomaz and Breazeal, 2006; Knox and Stone, 2009; Suay and Chernova, 2011; Knox and Stone, 2012; Griffith et al., 2013; Subramanian et al., 2016; Loftin et al., 2016; MacGlashan et al., 2017) has shown that RL agents can successfully speed up ”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “The agent that can ask for help from the demonstrator was proposed in [132].”
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks (2016, methodology): “Other metrics for curiosity have also been proposed, including prediction error [10, 35], prediction error improvement [36], and leverage [14].”