Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation

By Shreeyak S. Sajjan, Matthew Jackson Moore, Mike Pan, Ganesh Nagaraja, Johnny Lee, Andy Zeng, Shuran Song. IEEE International Conference on Robotics and Automation, 2019

309 citations recorded by the source corpus.

Transparent objects are a common part of everyday life, yet they possess unique visual properties that make them incredibly difficult for standard 3D sensors to produce accurate depth estimates for. In many cases, they often appear as noisy or distorted approximations of the surfaces that lie behind them. To address these challenges, we present ClearGrasp – a deep learning approach for estimating accurate 3D geometry of transparent objects from a single RGB-D image for robotic manipulation. Given a single RGB-D image of transparent objects, ClearGrasp uses deep convolutional networks to infer surface normals, masks of transparent surfaces, and occlusion boundaries. It then uses these outputs to refine the initial depth estimates for all transparent surfaces in the scene. To train and test ClearGrasp, we construct a large-scale synthetic dataset of over 50,000 RGB-D images, as well as a real-world test benchmark with 286 RGB-D images of transparent objects and their ground truth geometries. The experiments demonstrate that ClearGrasp is substantially better than monocular depth estimation baselines and is capable of generalizing to real-world images and novel objects. We also demonstrate that ClearGrasp can be applied out-of-the-box to improve grasping algorithms’ performance on transparent objects. Code, data, and benchmarks will be released. Supplementary materials: https://sites.google.com/view/cleargrasp1

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/de19cb46e9cf444a4fb9d80ea353ee5c8d7f86fe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains (2022, background): “Some recent papers [58], [59] focused on solving this problem with RGB information, but it is not fully addressed yet.”
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations (2019, background): “However, most prior self-supervised grasping systems are often constrained to learning in simulation [15], [17], [20], [24] or structured lab environments [1], [3], [21], [22].”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “Works on estimating transparent object pose and geometry might assume knowing object 3D model [20, 14] or rely on synthetic data to train vision models [24, 23].”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “2020), and some other methods (Wang et al. 2018; Yan et al. 2018, 2019; Gao and Tedrake 2019; Sajjan et al. 2019) utilize the RGB-D images.”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “PH in some applications. In addition to throwing, there are 3 other aspects that enable our system’s picking speeds: 1) fast algorithmic runtime speeds (220ms for inference), 2) real-time TSDF fusion [7,28,40,30] of RGB-D data, which enables us to capture and aggregate observed 3D data of the scene ”