Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations

By Shuran Song, Andy Zeng, Johnny Lee, Thomas A. Funkhouser. 2019

276 citations recorded by the source corpus.

Intelligent manipulation benefits from the capacity to flexibly control an end-effector with high degrees of freedom (DoF) and dynamically react to the environment. However, due to the challenges of collecting effective training data and learning efficiently, most grasping algorithms today are limited to top-down movements and open-loop execution. In this work, we propose a new low-cost hardware interface for collecting grasping demonstrations by people in diverse environments. This data makes it possible to train a robust end-to-end 6DoF closed-loop grasping model with reinforcement learning that transfers to real robots. A key aspect of our grasping model is that it uses “action-view” based rendering to simulate future states with respect to different possible actions. By evaluating these states using a learned value function (e.g., Q-function), our method is able to better select corresponding actions that maximize total rewards (i.e., grasping success). Our final grasping system is able to achieve reliable 6DoF closed-loop grasping of novel objects across various scene configurations, as well as in dynamic scenes with moving objects.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “Our idea of spatially-invariant action space is inspired by a series of work by Zeng et al. and colleagues [34, 37, 39–42].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “, logistics, agriculture, household) – leading to renewed interest in leveraging learned models to achieve general pick-and-place policies [23, 24, 25, 26, 27, 28, 29, 30] that can handle unseen objects.”
Visual Imitation Made Easy (2020, methodology): “This assistive tool is commonly used for grasping trash among other activities of daily living and has recently been shown to be a scalable interface for collecting grasping data in the wild by Song et al. [9].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “On the other hand, end-to-end perception-to-action models can learn precise sequential policies [2, 4, 6, 33, 34, 35], but these methods have limited understanding of semantic concepts and rely on goal-images to condition policies.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “While “pick-and-place” and “wipe” seem simple, they do cover a wide range of tabletop manipulation tasks and are crucial to industrial use cases like warehouse robots (Yoon et al., 2003; Berscheid et al., 2020; Devin et al., 2020; Song et al., 2019).”
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes (2021, background): “To address the limitations of planar grasping, there has been a recent interest in tackling the problem of 6-DoF grasping of unknown objects [10, 11, 12, 13, 14].”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, unknown): “The focus for the design of GELLO is to create an interface that is both economically accessible and easy-to-use for users aiming to render high-quality demonstrations in robot learning.”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Direct learning of pick-and-place policies meanwhile requires large amounts of data from demonstrations [2, 14, 40].”
Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019, background): “RGB-D images, point clouds) [50, 60, 33, 46], many of these algorithms cannot be immediately applied to transparent objects – which remain critical for applications like dish washing or sorting/cleaning plastic containers.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Unlike previous imitation learning studies with 2-DoF grippers [86, 75], we need the human video to guide the 30-DoF robot hand to move each finger in 3D space.”
AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains (2022, background): “[37] collected grasping demonstrations with a low-cost hand-held gripper.”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “While other works in 6-DoF grasping [30, 31, 32, 33, 34] have used RGB-D and pointcloud input, they have not been applied to sequential tasks or used with language-conditioning.”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, unknown): “Finally, there are some works that leverage cheap, off-the-shelf tools, such as reacher-grabber tools, for data collection, equipping robots with the same tools to allow for zero-shot transfer to the robot [47, 53, 63].”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, background): “Other projects have tried using low-cost, handheld grasping devices to collect large datasets [34, 35], but this technique requires inferring actions from video and then transferring behaviors to a robot.”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, background): “In contrast to the binary open-close action used in prior works [41, 44, 54], we found commanding gripper width continuously significantly expands the range of tasks doable by parallel-jaw grippers.”