Learning from Demonstrations for Real World Reinforcement Learning

By Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John P. Agapiou and 2 more. 2017

160 citations recorded by the source corpus.

Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages this data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator's actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 82 million steps to catch up to DQfD's performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 17 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.

Subfields: Artificial Intelligence, Economics and Econometrics, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Information Systems

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “, 2017), and bootstrapping RL from demonstrations (Silver et al., 2016; Hester et al., 2017).”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “, human) trajectories (Hester et al., 2017).”
Distributed Prioritized Experience Replay (2018, methodology): “As a result, prioritized experience replay is used in many agents, such as Prioritized Dueling DQN (Wang et al., 2016), UNREAL (Jaderberg et al., 2017), DQfD (Hester et al., 2017), and Rainbow (Hessel et al., 2017).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “…games, such as Bellemare et al.’s (2016b) success in Montezuma’s Revenge, none of these approaches has been able to improve, in a meaningful way, agents’ performance in games such as Pitfall!, where the only successes to date involve some form of apprenticeship (e.g., Hester et al., 2017).”
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards (2017, methodology): “The most similar work to ours is DQfD [3], which combines Deep Q Networks (DQN) [8] with learning from demonstrations in a similar way to DDPGfD.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “[23] T.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “Promising recent attempts at overcoming exploration in such tasks leverage demonstrations of successful trajectories (Kim et al., 2013; Hester et al., 2017; Vecerik et al., 2017; Nair et al., 2018; Gulcehre et al., 2020), dubbed the RL with Expert Demonstrations (RLED; Piot et al., 2014).”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “This approach has proven successful in domains where collecting such demonstrations is natural, such as simulated environments [24], [35], driving [4], [5], [6], and drones [36].”
Reinforcement Learning from Imperfect Demonstrations (2018, methodology): “Several previous efforts have shown that demonstrations can speed up RL by mimicking expert data with a temporal difference regularizer (Hester et al., 2017) or via gradient-free optimization (Ebrahimi et al.”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, methodology): “Our method is closest to two recent approaches — Deep Q-Learning From Demonstrations (DQfD) [34] and DDPG From Demonstrations (DDPGfD) [2] which combine demonstrations with reinforcement learning.”
StarCraft II: A New Challenge for Reinforcement Learning (2017, methodology): “The best results on this game were achieved by FullyConv LSTM and Random Search, while Atari-Net failed to learn a strategy to consistently produce marines during each episode.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “to use provided trajectories to guide the learning of suitable state-action pairs, but fine-tune the agent using RL [39].”
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces (2017, methodology): “Deep TAMER also performs better than the learning from demonstration method from (Hester et al. 2017), indicating an advantage of a reward shaping methods over a demonstration methods when the task is difficult for a human to perform but not difficult for a human to critique.”
AI Safety Gridworlds (2017, background): “…functions such as inverse reinforcement learning (Ng and Russell, 2000; Ziebart et al., 2008), learning from demonstrations (Abbeel and Ng, 2004; Hester et al., 2017), and learning from human reward feedback (Akrour et al., 2012; Wilson et al., 2012; MacGlashan et al., 2017; Christiano et al.,…”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “This resembles the training procedure adopted in recent game playing applications (Silver et al., 2016; Hester et al., 2017).”
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning (2018, background): “One approach to overcome exploration is by using human demonstrations [4, 5], but collecting human demonstrations remain challenging for many tasks, and it is difficult to scale.”