Learning Rich Features from RGB-D Images for Object Detection and Segmentation

By Saurabh Gupta, Ross B. Girshick, Pablo Arbeláez, Jitendra Malik. 2014

1,635 citations recorded by the source corpus.

In this paper we study the problem of object detection for RGB-D images using semantically rich image and depth features. We propose a new geocentric embedding for depth images that encodes height above ground and angle with gravity for each pixel in addition to the horizontal disparity. We demonstrate that this geocentric embedding works better than using raw depth images for learning feature representations with convolutional neural networks. Our final object detection system achieves an average precision of 37.3%, which is a 56% relative improvement over existing methods. We then focus on the task of instance segmentation where we label pixels belonging to object instances found by our detector. For this task, we propose a decision forest approach that classifies pixels in the detection window as foreground or background using a family of unary and binary tests that query shape and geocentric pose features. Finally, we use the output from our object detectors in an existing superpixel classification framework for semantic scene segmentation and achieve a 24% relative improvement over current state-of-the-art for the object categories that we study. We believe advances such as those represented in this paper will facilitate the use of perception in fields like robotics.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, methodology): “Our approach does not make use of pre- and post-processing complications, including superpixels [7, 15], proposals [15, 13], or post-hoc refinement by random fields or local classifiers [7, 15].”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “Accurate recognition and localisation of objects enables many applications, such as autonomous driving [8], image-editing [46] and robotics [16].”
Shape-aware Instance Segmentation (2016, background): “As such, it has many applications in diverse areas, such as autonomous driving [32], personal robotics [11] and plant analytics [27].”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “The methods proposed by Gupta et al. [13, 14] use 1.3M labeled images for supervised pre-training.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “various prediction tasks, including gaze estimation [43], text detection and classification in RGB images [9, 15], font recognition [42], object detection [10, 27], hand pose estimation in depth images [38, 37], scene recognition in RGB-D [11], semantic segmentation of urban scenes [31], and human p”
EDTER: Edge Detection with Transformer (2022, neutral): “In NYUDv2, we conduct experiments on RGB images and compare against the state-of-the-art methods including gPb-ucm [1], Silberman et al. [49], gPb+NG [19], SE [15], SE+NG+ [20], OEF [21], SemiCon-tour [69], HED [65], RCF [36], AMH-Net [66], LPCB [12], BDCN [22], and PiDiNet [53].”
Object-Proposal Evaluation Protocol is ‘Gameable’ (2015, methodology): “Object proposals are evaluated using this metric in two ways: – plotting Recall-vs-#proposals by ﬁxing t – plotting Recall-vs-t by ﬁxing the #proposals in L .”
Holistically-Nested Edge Detection (2015, methodology): “Finally following the experimental setup in [11] and [6], the maximum tolerance allowed for correct matches of edge predictions to ground truth during evaluation increases from .0075 to .011.”
Fast Edge Detection Using Structured Forests (2014, background): “Structured edges (SE) and SE coupled with depth normal gradient (SE þ NGþ) [21], [22] achieve top results.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, methodology): “In the 40-class task, our model outperforms Gupta et al. ’14 with both model sizes, and Long et al. with the larger size.”
ParseNet: Looking Wider to See Better (2015, background): “Attributed to the large learning capacity of deep neural networks [15, 27, 26], convolutional neural networks (CNN) have become a powerful tool not only for whole image classification, but also for object detection and semantic segmentation [4, 27, 28, 9, 6].”
Cross-Stitch Networks for Multi-task Learning (2016, background): “, an image as opposed to say an image and a depth-map [25].”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, methodology): “Our method signiﬁ-cantly outperforms the existing methods. method training data pixel accuracy mean accuracy IoU Gupta et al. [16] RGB-D 60.3 - 28.6 FCN-32s [32] RGB 60.0 42.2 29.2 FCN-HHA [32] RGB-D 65.”
Adversarial Discriminative Domain Adaptation (2017, methodology): “We consider the task of adaptation between these RGB and HHA encoded depth images [28], using them as source and target domains respectively.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “For the networks trained with only RGB data, we follow [61,65], which directly use them to extract feature on the depth representation.”
RGB-D datasets using microsoft kinect or similar sensors: a survey (2016, methodology): “The latest algorithm performance comparisons based on B3DO, Person Re-identification, Kinect FaceDB, Big BIRD and HRRFaceD can be found in [32, 40, 64, 66] and [62] respectively.”
Cross Modal Distillation for Supervision Transfer (2015, methodology): “Acknowledgments: The authors would like to thank Georgia Gkioxari for sharing her wisdom and experimental setup for the UCF 101 and JHMDB datasets.”
Volumetric and Multi-view CNNs for Object Classification on 3D Data (2016, background): “Impressive performance for object detection from RGB-D images has been achieved using a geocentric embedding for depth images that encodes height above ground and angle with gravity for each pixel in addition to the horizontal disparity [11].”