Deep Q-learning From Demonstrations

By Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband and 4 more. 2017

1,289 citations recorded by the source corpus.

Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages small sets of demonstration data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator’s actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 83 million steps to catch up to DQfD’s performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 11 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “…learning methods to large deep learning systems, for example inverse RL (Finn et al., 2016), imitation learning (Ho and Ermon, 2016; Stadie et al., 2017), semi-supervised skill generalization (Finn et al., 2017), and bootstrapping RL from demonstrations (Silver et al., 2016; Hester et al., 2017).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “It is generally difﬁcult to outperform humans using imitation learning alone (Hester et al., 2018): even a perfect imitator can only perform as well as the source it is imitating; superhuman performance typically comes from executing human action sequences faster and more reliably by smoothing out…”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “We also accidentally noticed that DQfD is extremely sensitive to small differences between the demonstration observations and the agent’s experience—a small misalignment in the score-blanking method between demos and policy frames reduced the agent’s scores to zero in the demo-heavy games.”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “The noise levels ε i are chosen as Notably, this exploration is closer to the one used by Hester et al. [7] and is much lower ( i.e., less random exploration) than the schedule used by Horgan et al. [8].”
Experience Replay for Continual Learning (2018, background): “Research in this vein has included prioritized replay for maximizing the impact of rare experiences [28], learning from human demonstration data seeded into a buffer [9], and methods for approximating replay buffers with generative models [30].”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Prior work has explored some forms of Q-function penalties [23, 58], but only in the standard online RL setting with demonstrations.”
Causal Reasoning from Meta-reinforcement Learning (2019, methodology): “…in this direction, obtained using relatively standard deep RL components – our approach will likely benefit from more advanced architectures (e.g. Hester et al., 2017; Hessel et al., 2018; Espeholt et al., 2018) that allow us to train on longer more complex episodes, as well as models which…”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “require further interactions with the environment to compensate (Hester et al., 2017; Sun et al., 2018; Cheng et al., 2018).”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, positive): “This kind of large margin loss has been shown to be effective in practice by previous works like DQfD [12].”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, neutral): “DQfD [69] pre-trains the agent and uses expert demonstrations by adding them into the replay buffer with additional priority.”
Self-Imitation Learning (2018, background): “Learning from imperfect demonstrations A few studies have attempted to learn from imperfect demonstrations, such as DQfD (Hester et al., 2018), Q-ﬁlter (Nair et al., 2017), and normalized actor-critic (Xu et al., 2018).”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “…as discussed in Section 4, KL-control [22] (which solves a KL-penalized RL problem similarly to maximum entropy RL), a static version of DQfD [21] (where a constraint to upweight Q-values of state-action pairs observed in the dataset is added as an auxiliary loss on top a regular…”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “, 2018), and a common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function (Rajeswaran et al., 2017; Nair et al., 2018; Večerík et al., 2017; Hester et al., 2018; Gao et al., 2018).”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…in a variety of manners, such as mixed with adversarial methods [Zhu et al., 2018, Kang et al., 2018], used for pre-training [Pfeiffer et al., 2018], modifying the replay buffer [Večerík et al., 2017, Gulcehre et al., 2020] or adjusting the value function [Kim et al., 2013, Hester et al., 2017].”
Hyperparameter Selection for Imitation Learning (2021, background): “Although expert trajectories can also be useful in this case, this is not the setting of IL, but of RL with demonstrations where both the expert demonstrations and reward signals are used (Kim et al., 2013; Piot et al., 2014; Hester et al., 2018).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “, 2018), learning from demonstrations (Hester et al., 2018; Nair et al., 2018; Gulcehre et al., 2020), variance reduction (Wang et al.”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “Imitation and Reinforcement Learning from Suboptimal Demonstrations : Recent work has tried to leverage off-policy deep reinforcement learning in conjunction with a set of demonstrations to account for suboptimal data and learn policies that outperform the demonstrations [12, 14, 28, 36, 38].”