A textured object recognition pipeline for color and depth image data

By Jie Tang, Stephen Miller, Arjun Singh, Pieter Abbeel. 2012

141 citations recorded by the source corpus.

We present an object recognition system which leverages the additional sensing and calibration information available in a robotics setting together with large amounts of training data to build high fidelity object models for a dataset of textured household objects. We then demonstrate how these models can be used for highly accurate detection and pose estimation in an end-to-end robotic perception system incorporating simultaneous segmentation, object classification, and pose fitting. The system can handle occlusions, illumination changes, multiple objects, and multiple instances of the same object. The system placed first in the ICRA 2011 Solutions in Perception instance recognition challenge. We believe the presented paradigm of building rich 3D models at training time and including depth information at test time is a promising direction for practical robotic perception systems.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Most successful approaches rely on using multiple camera frames [6] or depth information [44].”
Deep Object-Centric Representations for Generalizable Robot Learning (2017, methodology): “manipulation, SIFT features have also been used for 3D pose estimation and object localization, using object-specific training data gathered individually for the task [23], [24].”
BigBIRD: A large-scale 3D database of object instances (2014, background): “Though the advent of commodity RGB-D sensors, such as the Microsoft Kinect, aid in addressing 3D pose detection and localization by providing a depth channel in addition to a color channel, instance recognition systems still cannot reliably detect hundreds of objects [1], [2], [3].”
Deep Learning Advances in Computer Vision with 3D Data (2017, methodology): “On the other hand, global descriptors-based methods (e.g., Rusu et al. [2010], Aldoma et al. [2012b], Tang et al. [2012], and Wang et al. [2013]) employ global descriptors and require segmentation of the scene in candidate objects, since only one descriptor is computed for each input point cloud.”
Cross-modal adaptation for RGB-D detection (2016, background): “This has motivated the use of depth information from commodity RGB-D sensors to improve object recognition performance [20], [19], [32], [31], [47].”
3D ShapeNets for 2.5D Object Recognition and Next-Best-View Prediction (2014, background): “Even though there are many theories about 3D representation [4, 18], the success of 3D-based methods is largely limited to instance recognition, using model-based keypoint matching [20, 24].”
Multiview RGB-D Dataset for Object Instance Detection (2016, background): “[30] take advantage of the discriminative nature of the local descriptors.”
50 Years of object recognition: Directions forward (2013, background): “[296] achieved top ranked performance in a related recognition challenge, by leveraging the ability of the Kinect to provide accurate depth information in order to build reliable 3D object models.”
3D ShapeNets: A deep representation for volumetric shapes (2014, unknown): “modelbased keypoint matching to nearest neighbors [24, 31]).”
Efficient Multi-cue Scene Segmentation (2013, unknown): “[13, 24].”
Unsupervised Feature Learning for RGB-D Based Object Recognition (2012, methodology): “Extensive evaluations on several publicly available benchmark datasets [19, 9, 35] allowed us to gain various experimental insights: unsupervised feature learning from raw data can yield recognition accuracy that is superior to state-of-the-art object recognition algorithms, even to ones specificall”
Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd (2015, background): “When objects have sufficient texture, techniques based on key-point matching [22, 30] demonstrate good results, yet when there is a lot of clutter in the scene they depict many false positive matches which degrades their performance.”
Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images (2013, background): “Subsequently, there have also been numerous papers in both robotics and vision communities looking at object and instance recognition using RGB-D data [28, 17] and dense reconstruction of indoor scenes and objects from multiple scenes.”
Open-vocabulary Object Retrieval (2014, methodology): “In such scenarios, humans rarely name an object with a single basic-level noun (e.g., “box”).”
Learning hierarchical sparse features for RGB-(D) object recognition (2014, methodology): “…accuracy that is superior to state-of-the-art object recognition algorithms, even to ones speciﬁcally designed and tuned for textured objects (Tang et al., 2012); HMP can take full advantage of the additional information contained in color and depth channels and signiﬁcantly boosts the…”
Second-order constrained parametric proposals and sequential search-based structured prediction for semantic segmentation in RGB-D images (2015, background): “estimation[42], Kinect has also spurred a wave of scene understanding research in robotics[39, 45] and computer vision[44, 39, 18, 34, 2, 3] with datasets[43, 28] recently made available.”
Sliding Shapes for 3D Object Detection in Depth Images (2014, background): “Since our generic object detector does not reply on any assumption about the background or requires a dominant supporting plane [15, 16], and its single-view nature doesn’t require a (semi-)complete scan of an object [17], it can be used as a basic building block for general scene understanding task”