Making Efficient Use of Demonstrations to Solve Hard Exploration Problems

By T. Paine, Caglar Gulcehre, Bobak Shahriari, Misha Denil, Matthew D. Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil C. Rabinowitz, Duncan Williams and 4 more. 2019

97 citations recorded by the source corpus.

This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…in a variety of manners, such as mixed with adversarial methods [Zhu et al., 2018, Kang et al., 2018], used for pre-training [Pfeiffer et al., 2018], modifying the replay buffer [Večerík et al., 2017, Gulcehre et al., 2020] or adjusting the value function [Kim et al., 2013, Hester et al., 2017].”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “Imitation learning algorithms such as R2D3 (Paine et al., 2019) relate to the oﬄine RL algorithms called growing-batch RL algorithms which grows the oﬄine datasets with environment interactions.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “, 2019), Hard-Eight (Paine et al., 2019) and learning motor control for high-dimensional simulated robots (Heess et al.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “, 2018), learning from demonstrations (Hester et al., 2018; Nair et al., 2018; Gulcehre et al., 2020), variance reduction (Wang et al.”
Using Unity to Help Solve Intelligence (2020, methodology): “• Making Eﬃcient Use of Demonstrations to Solve Hard Exploration Problems (Paine et al., 2019): Introduces a suite of eight hard exploration, ﬁrst-person, 3D tasks with highly variable initial conditions.”
TextGAIL: Generative Adversarial Imitation Learning for Text Generation (2020, methodology): “We apply GAIL to text generation, and use proximal policy optimization (PPO) (Schulman et al., 2017), and efﬁcient use of human demonstrations (Paine et al., 2019) to further stabilize the training.”
What Matters for Adversarial Imitation Learning? (2021, background): “When demonstrations as well as external rewards are available, it is common for RL algorithms to sample batches for off-policy updates from the demonstrations in addition to the replay buffer [30, 39].”