Shape completion enabled robotic grasping

By Jacob Varley, Chad DeChant, Adam Richardson, J. Ruales, Peter K. Allen. IEEE/RJS International Conference on Intelligent RObots and Systems, 2016

349 citations recorded by the source corpus.

This work provides an architecture to enable robotic grasp planning via shape completion. Shape completion is accomplished through the use of a 3D convolutional neural network (CNN). The network is trained on our own new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. At runtime, a 2.5D pointcloud captured from a single point of view is fed into the CNN, which fills in the occluded regions of the scene, allowing grasps to be planned and executed on the completed object. Runtime shape completion is very rapid because most of the computational costs of shape completion are borne during offline training. We explore how the quality of completions vary based on several factors. These include whether or not the object being completed existed in the training data and how many object models were used to train the network. We also look at the ability of the network to generalize to novel objects allowing the system to complete previously unseen objects at runtime. Finally, experimentation is done both in simulation and on actual robotic hardware to explore the relationship between completion quality and the utility of the completed mesh model for grasping.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/1863d67b653090151cd67daac7a635f3e1034a54

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers (2021, background): “Early attempts on point cloud completion [6, 15, 30, 31, 23, 36, 20, 19, 51, 47, 40] try to migrate mature methods from 2D completion tasks to 3D point clouds by voxelization and 3D convolutions.”
3D Shape Perception from Monocular Vision, Touch, and Shape Priors (2018, neutral): “[11] explored how to better grasp an object by ﬁrst employing a convolutional neural net for shape completion.”
TopNet: Structural Point Cloud Decoder (2019, background): “The ability to infer the full shape of an object from an incomplete scan acquired by depth camera or LiDAR is indeed an important task which enables several downstream applications such as robotics manipulation [31], scene understanding for navigation [7], and virtual manipulation of completed shape”
Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network (2019, methodology): “[15] propose an interesting approach to grasp planning through shape completion where a 3D CNN was used to train the network on the 3D prototype of objects on their own dataset captured from various viewpoints.”
Learning Grasping Interaction with Geometry-aware 3D Representations (2017, background): “…grasping planning and control often involves 3D modeling of object shape, modeling dynamics of robot hands, and local surface modeling Goldfeder et al. [2009], Leon et al. [2010], Johns et al. [2016], Varley et al. [2016], Li et al. [2016], Vahrenkamp et al. [2016], Mahler et al. [2016, 2017].”
3D Object Reconstruction from a Single Depth View with Adversarial Learning (2017, methodology): “While state-of-the-art deep learning approaches [7] [61] [6] [58] [62] for 3D shape reconstruction achieve encouraging and compelling results, they are limited to a very small resolution, typically less than 40(3) voxel grids.”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “used a fully connected deep network to predict candidate fingertip locations on depth images for known 3D object models and later extended the method [59] to estimate the object shape from a single-view point cloud using a 3D CNN.”
PointNetGPD: Detecting Grasp Configurations from Point Sets (2018, methodology): “Motivated by the success of deep neural networks in various 3D computer vision tasks [21]–[24], several trials on combining 3D computer vision techniques and grasp planning have been carried out [3], [4], [8], [9], [12].”
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis (2017, background): “In robotics, synthesized novel views give the robot a better understanding of unseen parts of the object through 3D reconstruction, which will be helpful for grasp planning [41].”
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View (2017, background): “Recently there have been many works addressing the problem of shape completion for individual objects [35, 26, 38] or scenes [36, 33, 8].”
Anisotropic Convolutional Networks for 3D Semantic Scene Completion (2020, background): “An open topic in computer vision is to study how to enable machines such an ability, which is desirable in many applications such as navigation [4], grasping [20], 3D home design [1], to name a few.”
Deformable Shape Completion with Graph Convolutional Autoencoders (2017, methodology): “While several methods based on (volumetric) convolutional neural networks have been proposed for completing man-made rigid objects (see [12, 51, 55, 61, 49]), they struggle at handling deformable shapes.”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “In [2], a network was designed to complete the 3D shape from a single 2.”
PCN: Point Completion Network (2018, methodology): “While most existing learning-based methods [9, 16, 44, 47, 50, 54, 56, 59] represents shapes using voxels, which are convenient for convolutional neural networks, our method uses point clouds, which preserve complete geometric information about the shapes while being memory efﬁcient.”
High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference (2017, methodology): “Recently, a number of methods [23, 28, 8] attempt to perform shape completion at a coarse resolution ( 32 3 ) us-ing 3DCNNs.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “…which has been recently addressed using deep learning techniques, see the recent survey of Laga [153], 3D shape completion [26], [28], [45], [103], [156], [160], [161], 3D reconstruction from depth images [103], which can be seen as a 3D fusion and completion problem, 3D reconstruction and…”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, positive): “For example, CLIPORT [165] and Transporter Networks [223] utilized Pybullet simulator data for training and successfully transferred models to real-world applications.”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, methodology): “Since the convolution can not be directly applied to point clouds due to their irregularity and unorderedness, most of the existing methods [2,3,4,5,6,7,8] voxelize the point cloud into binary voxels, where 3D convolutional neural networks can be applied.”
Semantic Scene Completion from a Single Depth Image (2016, background): “Other prior works focus on single object shape completion [33, 28, 34, 32].”