Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

By Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Manfred Otto Heess, Thomas Rothörl, Thomas Lampe, Martin A. Riedmiller. 2017

764 citations recorded by the source corpus.

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, methodology): “Večerík et al. (2017) apply the same technique to DDPG in robotics, and Zhang and Ma (2018) pretrain actor-critic architectures with demonstrations.”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “In robotics tasks, similar techniques have been combined with other improvements to successfully solve difficult exploration problems [14, 21].”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Prior work has explored some forms of Q-function penalties [23, 58], but only in the standard online RL setting with demonstrations.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “Note that “SAC (scratch),” which does not receive the prior data, performs similarly to “SACfD (prior),” which does have access to the prior data, indicating that the off-policy RL algorithm is not actually able to make use of the off-policy data for pre-training.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “In recent years, combining imitation with reinforcement learning, via learning from demonstrations (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015) has grown in popularity with extensions to deep reinforcement learning (Hester et al., 2017; Veˇcer´ık et al., 2017).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “, Atari [29]), this is significantly more challenging in real-world physical tasks, as it requires human intervention in the form of manual resets or the availability of a hard coded reset [28, 47, 37, 52, 21].”
A Framework for Data-Driven Robotics (2019, methodology): “As in prior works [24]–[26], we use demonstrations as part of the agent experience and train with temporal difference learning in a model-free setting.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…et al., 2013, Chemali and Lazaric, 2015, Piot et al., 2014], and a common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function [Rajeswaran et al., 2017, Nair et al., 2018, Večerík et al., 2017, Hester et al., 2018, Gao et al., 2018].”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “The work most closely related to ours is DDPGfD [53], where demonstrations are incorporated into DDPG [29] by adding them to the replay buffer.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…in a variety of manners, such as mixed with adversarial methods [Zhu et al., 2018, Kang et al., 2018], used for pre-training [Pfeiffer et al., 2018], modifying the replay buffer [Večerík et al., 2017, Gulcehre et al., 2020] or adjusting the value function [Kim et al., 2013, Hester et al., 2017].”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Promising recent attempts at overcoming exploration in such tasks leverage demonstrations of successful trajectories (Kim et al., 2013; Hester et al., 2017; Vecerik et al., 2017; Nair et al., 2018; Gulcehre et al., 2020), dubbed the RL with Expert Demonstrations (RLED; Piot et al., 2014).”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, positive): “learn policies that outperform the demonstrations [12, 14, 28, 36, 38].”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “Prior works show that data-driven IL methods improve both the sample efﬁciency of policy learning [16, 17, 18] and the performance of the trained policy [19, 20, 21].”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “community [17], [16], [18], [19], [1], [20], [21] but these methods have not been applied to real world manipulation with multi-fingered hands.”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, methodology): “Our method is closest to two recent approaches — Deep Q-Learning From Demonstrations (DQfD) [34] and DDPG From Demonstrations (DDPGfD) [2] which combine demonstrations with reinforcement learning.”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Deep reinforcement learning (RL) has made significant progress on a range of continuous control tasks, such as locomotion skills [35, 25, 17], learning dexterous manipulation behaviors [33], and training robot arms for simple manipulation tasks [13, 41].”