Overcoming Exploration in Reinforcement Learning with Demonstrations

By Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel. 2017

892 citations recorded by the source corpus.

Exploration in environments with sparse rewards has been a persistent problem in reinforcement learning (RL). Many tasks are natural to specify with a sparse reward, and manually shaping a reward function can result in suboptimal performance. However, finding a non-zero reward is exponentially more difficult with increasing task horizon or action dimensionality. This puts many real-world tasks out of practical reach of RL methods. In this work, we use demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm. Our method, which builds on top of Deep Deterministic Policy Gradients and Hindsight Experience Replay, provides an order of magnitude of speedup over RL on simulated robotics tasks. It is simple to implement and makes only the additional assumption that we can collect a small set of demonstrations. Furthermore, our method is able to solve tasks not solvable by either RL or behavior cloning alone, and often ends up outperforming the demonstrator policy.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, methodology): “Nair et al. (2018) combine these methods with hindsight experience replay (Andrychowicz et al., 2017).”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “In robotics tasks, similar techniques have been combined with other improvements to successfully solve difﬁcult exploration problems [14, 21].”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “For example, prior methods that incorporate demonstrations into RL directly aim to mimic these demonstrations [11], which is desirable when the demonstrations are known to be optimal, but can cause undesirable bias when the prior data is not optimal.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Research in imitation, and inverse reinforcement learning, with robustness to noise is an emerging area (Evans, 2016; Nair et al., 2018), but relies on some form of expert data.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Learning : Several recent works (Veˇcer´ık et al., 2017; Nair et al., 2017; Gao et al., 2018; Zhu et al., 2018) have introduced algorithms that unify reinforcement learning and imitation learning such that the learning agent can beneﬁt from both expert demonstrations and interactions with the…”
Self-Imitation Learning (2018, background): “Learning from imperfect demonstrations A few studies have attempted to learn from imperfect demonstrations, such as DQfD (Hester et al., 2018), Q-ﬁlter (Nair et al., 2017), and normalized actor-critic (Xu et al., 2018).”
A Framework for Data-Driven Robotics (2019, background): “Demonstrations: Demonstrations in RL have gained popularity in recent years [22], [23] as they help to address the exploration problem.”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “…with the relabeled demonstration pools D l and D h from Section 4.2, as described in Eqn 2 and 3: While a similar objective has been described in [28, 30], it is yet to be explored in the hierarchical, goal-conditioned scenarios, which makes a signiﬁcant difference as indicated in our experiments.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “18 [Nair et al., 2018] We use the implementation from https://github.com/ jangirrishabh/Overcoming-exploration-from-demos.”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “In order to benchmark the capabilities of DRL with regard to the dexterous manipulation tasks outlined in Section III, we evaluate the NPG algorithm described briefly in Section V, and the DDPG algorithm [29], which has recently been used in a number of robotic manipulation scenarios [53], [13], [32”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…from demonstrations [Atkeson and Schaal, 1997]), BC has been used as a regularization for policy optimization with DDPG [Lillicrap et al., 2015, Nair et al., 2018, Goecks et al., 2020] and the natural policy gradient [Kakade, 2001, Rajeswaran et al., 2017], but with additional sophistication…”
Critic Regularized Regression (2020, neutral): “A simple (binary) version of the ﬁltering component of CRR was proposed by Nair et al. [26], whose results showed that the ﬁlter simply accelerated training, but gave mixed results in long horizons.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “Promising recent attempts at overcoming exploration in such tasks leverage demonstrations of successful trajectories (Kim et al., 2013; Hester et al., 2017; Vecerik et al., 2017; Nair et al., 2018; Gulcehre et al., 2020), dubbed the RL with Expert Demonstrations (RLED; Piot et al., 2014).”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, positive): “learn policies that outperform the demonstrations [12, 14, 28, 36, 38].”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “This simple method for doing reinforcement learning with demonstrations has been motivated theoretically by [45] and used in practice in prior and concurrent work [46, 47, 39].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “In comparison, single-task methods for learning from demonstrations and rewards typically require hundreds or thousands of trials to learn tasks of comparable difficulty [29, 26].”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “community [17], [16], [18], [19], [1], [20], [21] but these methods have not been applied to real world manipulation with multi-fingered hands.”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “Nair et al. [30] extends the DDPGfD algorithm [49] to learn a block stacking task on a position-controlled arm in simulation.”
Sim-to-Real Reinforcement Learning for Deformable Object Manipulation (2018, background): “Behavioural Cloning [28] DDPG can be further adapted to take advantage of demonstrations by introducing behavioural cloning loss to the actor network.”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, background): “…policy learning from demonstration has been previously studied under the umbrella of accelerating reinforcement learning by structured exploration (Nair et al., 2017; Hester et al., 2017) An implicit assumption of these approaches is access to demonstrations and reward from the environment; our…”