Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning

By Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, Thomas A. Funkhouser. 2018

654 citations recorded by the source corpus.

Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end-effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors even amid challenging cases of tightly packed clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu

Subfields: Reinforcement Learning, Representation & Self-Supervised Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Representation & Self-Supervised Learning

In Robotics & Control

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…prior works have explored offline RL methods for learning robotic grasping, which is a particularly appealing task offline RL methods, since it requires the ability to generalize to a wide range of objects (Pinto and Gupta, 2016; Levine et al., 2018; Kalashnikov et al., 2018; Zeng et al., 2018).”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Deep neural network models have been used widely in a range of robotics applications [12, 13, 14, 15, 16, 17].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “A number of recent works have studied self-supervised robotic learning, where large-scale unattended data collection is used to learn individual skills such as grasping [22], [23], [24], [25], push-grasp synergies [26], or obstacle avoidance [27], [28].”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “[5] recently proposed a Q-learning framework for combining grasping and pushing.”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, background): “, grasping and pushing [40]) at that spatial location.”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, methodology): “Spatial equivariance has previously been shown to improve learning efficiency for vision-based picking [29, 30, 31, 32, 34].”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, methodology): “• Joint learning of grasping and throwing policies with a deep neural network that maps from visual observations (of objects in a bin) to control grasping and throwing parameters: the likelihood of grasping success for a dense pixelwise sampling of end effector orientations and locations [33], and t”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “State-of-the-art grasping systems typically either operate in an open-loop style, where grasping locations are chosen, and then a motion is executed to complete the grasp [69, 41, 37, 60], or in a closed-loop manner, where grasp prediction is continuously run during motion, either explicitly [65], o”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, methodology): “RGB-D sensors can be beneficial [1], [11], [12], as they capture physical information about the workspace.”
Learning ambidextrous robot grasping policies (2019, methodology): “The policy uses an instance recognition system to match object segments to previous failures in a database ( 12 ) and pushes objects to create accessible grasps when none are available (see Materials and Methods for details on the memory-based policy).”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Many works frame grasping as a bandit or classification problem, where the action space consists of discrete grasp candidates and the picking motion is executed open-loop (108, 109).”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “grasping, pushing or placing) to each location [111, 131, 132].”
6-DOF GraspNet: Variational Grasp Generation for Object Manipulation (2019, methodology): “Apart from using supervised learning, grasping has also been formulated as a reinforcement learning problem [9, 36] or approximations of it [14].”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “free RL approaches have illustrated the ability to generalize to new objects (Kalashnikov et al., 2018) and learn tasks such as grasping and pushing through self-supervision (Pinto & Gupta, 2015; Zeng et al., 2018), pure model-free approaches generally lack the ability to explicitly reason over ”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “RL has been applied to a wide variety of robotic manipulation tasks, including grasping objects [11, 12], in-hand object manipulation [13, 14, 15, 16], pouring fluids [17], door opening [18, 19], and manipulating cloth [20, 21].”
Deictic Image Maps: An Abstraction For Learning Pose Invariant Manipulation Policies (2018, methodology): “For example, (Li, Hsu, and Lee 2018) uses a hand-coded heuristic to prune action choices and (Zeng et al. 2018) uses a fully convolutional network architectures to evaluate Q-values for millions of actions.”
Self-Supervised Exploration via Disagreement (2019, background): “Following (Zeng et al., 2018), instead of adding the 16 rotations in the output, we pass 16 equally spaced rotated images to the network and then sample actions based on the output of all the inputs.”
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations (2019, background): “This pregrasp approaching process is a time-varying sequence of actions, for which rewards are loosely defined, and has previously been shown to be more effectively learned through reinforcement than from direct supervision [3], [19].”