Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods

By Deirdre Quillen, Eric Jang, Ofir Nachum, Chelsea Finn, Julian Ibarz, Sergey Levine. 2018

219 citations recorded by the source corpus.

In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best suited for a rich, diverse task like grasping. To answer this question, we propose a simulated benchmark for robotic grasping that emphasizes off-policy learning and generalization to unseen objects. Off-policy learning enables utilization of grasping data over a wide variety of objects, and diversity is important to enable the method to generalize to new objects that were not seen during training. We evaluate the benchmark tasks against a variety of Q-function estimation methods, a method previously proposed for robotic grasping with deep neural network models, and a novel approach based on a combination of Monte Carlo return estimation and an off-policy correction. Our results indicate that several simple methods provide a surprisingly strong competitor to popular algorithms such as double Q-learning, and our analysis of stability sheds light on the relative tradeoffs between the algorithms 11Accompanying video: https://goo.gl/pyMd6p.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, methodology): “The image s and action a are inputs into our network, and the arg max in Equation (1) is evaluated by means of a stochastic optimization algorithm that can handle non-convex and multimodal optimization landscapes, similarly to [19] and [20].”
Sim-to-Real Reinforcement Learning for Deformable Object Manipulation (2018, background): “This has been extensively studied in the context of rigid object manipulation (see [14] for a comprehensive evaluation), but only a small amount of work has focused on deformable objects.”
Off-Policy Evaluation via Off-Policy Classification (2019, methodology): “The discounted sum of advantages ( P P Finally, the Monte Carlo corrected error (MCC Error) is derived by arranging the discounted sum of advantages into a squared error, and was used as a training objective by Quillen et al. [29].”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, methodology): “Representation learning has been studied for executing speciﬁc tasks such as grasping [7, 8, 10], dexterous manipulation [26, 27], or continuous ﬂight [9].”
Q-Learning in enormous action spaces via amortized approximate maximization (2020, methodology): “Kalashnikov et al. (2018) and Quillen et al. (2018) used the cross-entropy method to perform an approximate maximization over actions in order to apply Q-learning to robotic grasping tasks.”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Prior, representation learning was studied on language tasks restricted to simulated discrete [10, 11, 12, 13, 14, 15] and continuous [16, 17, 18, 19] environments, or non-language robotic tasks [20, 21, 22, 23].”
CAQL: Continuous Action Q-Learning (2019, background): “Empirical results also suggest that value-based methods are more data-efficient and less sensitive to hyper-parameters (Quillen et al., 2018).”
Self-Supervised Learning for Precise Pick-and-Place Without Object Model (2020, methodology): “From our point of view, two fundamental approaches have emerged: First, an end-to-end approach using a step-wise, velocity-like control of the end effector [7]–[9].”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, background): “update its strategy while accumulating new observations [16], [37], [28].”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, background): “These techniques have learned to play Atari games beyond human-level [60], Go, chess, and shogi from scratch [82, 81], complex behaviors in 3D environments [59, 97, 37], robotics [27, 73], character animation [67], among others.”