Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images

By Shuran Song, Jianxiong Xiao. 2015

709 citations recorded by the source corpus.

We focus on the task of amodal 3D object detection in RGB-D images, which aims to produce a 3D bounding box of an object in metric form at its full extent. We introduce Deep Sliding Shapes, a 3D ConvNet formulation that takes a 3D volumetric scene from a RGB-D image as input and outputs 3D object bounding boxes. In our approach, we propose the first 3D Region Proposal Network (RPN) to learn objectness from geometric shapes and the first joint Object Recognition Network (ORN) to extract geometric features in 3D and color features in 2D. In particular, we handle objects of various sizes by training an amodal RPN at two different scales and an ORN to regress 3D bounding boxes. Experiments show that our algorithm outperforms the state-of-the-art by 13.8 in mAP and is 200× faster than the original Sliding Shapes.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, methodology): “Since then, the frameworks of RPN and Faster R-CNN have been adopted and generalized to other methods, such as 3D object detection [13], part-based detection [14], instance segmentation [15], and image captioning [16].”
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection (2017, methodology): “Recently, region proposal networks [34] have become an important building block of top-performing object detection frameworks [40, 5, 23].”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative [45,55,63] and generative [9,23,58,64,75,77] 3D tasks.”
Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, background): “Second, challenging in-context and partial observations of real-world objects are common due to occlusions and reconstruction errors; for example, they can be found in window-based object detectors [38] in many robotics or autonomous vehicle applications.”
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, background): “Thus, many of the current RGB-D datasets [71, 89, 74, 30] are orders of magnitude smaller than their 2D counterparts.”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “Increasing the resolution naı̈vely [32, 40, 52] reduces the depth of the networks and hence their expressiveness.”
Multi-view 3D Object Detection Network for Autonomous Driving (2016, methodology): “Deep Sliding Shapes [19] exploits more powerful deep learning features.”
SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis (2017, methodology): “Deep learning has also been successfully applied to other 3D applications like volumetric shape retrieval or object classification [28, 25, 16, 19].”
Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, background): “Since then different versions of 3D-CNN architectures have been proposed in order to improve classiﬁcation accuracy [22, 31, 34], obtain object proposals [37], or match local 3D geometry [46].”
Semantic Scene Completion from a Single Depth Image (2016, background): “To solve this problem we ﬂip the TSDF value (a) Object centric networks [34, 20] (b) 3DMatch [36] (c) Deep Sliding Shape [31] (d) SSCNet This ﬂipped TSDF has the strong gradient on surface, providing a more meaningful signal for the network to learn better geometric features.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “These previous RGB-D datasets have been used to train models for several standard scene understanding tasks, including semantic segmentation [27, 33, 13, 40, 7], 3D object detection [31, 24, 14, 36, 37], normal estimation [41, 23, 9, 3], camera relocalization [30, 39], and others [47, 11, 12].”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, background): “Song et al. [32] focus on 3D object detection in RGB-D scenes.”
PIXOR: Real-time 3D Object Detection from Point Clouds (2018, neutral): “With the popularity of inexpensive RGB-D sensors such as Microsoft Kinect, Intel RealSense and Apple PrimeSense, several approaches that utilize depth information and fuse them with RGB images have been developed [32, 33].”
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (2018, methodology): “proposed Sliding Shapes to predict 3D object bounding boxes from single RGB-D frame input with handcrafted feature design [30], and then expanded the approach to operate on learned features [31].”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “Deep Sliding Shapes Song and Xiao (2016) proposed the first 3D Region Proposal Network (RPN) using 3D convolutional neural networks (ConvNets) and the first joint Object Recognition Network (ORN) to extract geometric features in 3D and color features in 2D to regress 3D bounding boxes.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, background): “further extended this idea with 3D discrete convolutional neural networks [80].”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, methodology): “In [12] they propose a 3D CNN by replacing 2D projections with a 3D volumetric representation (projective D-TSDF volumes [40]).”
Joint 3D Proposal Generation and Object Detection from View Aggregation (2017, methodology): “space. We use the regressed orientation vector to resolve the ambiguity in the Fig. 4: A visual comparison between the 8 corner box encoding proposed in [4], the axis aligned box encoding proposed in [16], and our 4 corner encoding. bounding box orientation estimate from the adopted four corner repr”
Depth-aware CNN for RGB-D Segmentation (2018, background): “Alternatively, many works [11,23] attempt to solve the problem with 3D CNNs.”