Multimodal deep learning for robust RGB-D object recognition

By Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin A. Riedmiller, Wolfram Burgard. 2015

684 citations recorded by the source corpus.

Robust object recognition is a crucial ingredient of many, if not all, real-world robotics applications. This paper leverages recent progress on Convolutional Neural Networks (CNNs) and proposes a novel RGB-D architecture for object recognition. Our architecture is composed of two separate CNN processing streams - one for each modality - which are consecutively combined with a late fusion network. We focus on learning with imperfect sensor data, a typical problem in real-world robotics tasks. For accurate learning, we introduce a multi-stage training methodology and two crucial ingredients for handling depth data with CNNs. The first, an effective encoding of depth information for CNNs that enables learning without the need for large depth datasets. The second, a data augmentation scheme for robust learning with depth images by corrupting them with realistic noise patterns. We present state-of-the-art results on the RGB-D object dataset [15] and show recognition in challenging RGB-D real-world noisy settings.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Volumetric and Multi-view CNNs for Object Classification on 3D Data (2016, background): “Recently, a CNN architecture has been proposed where the RGB and depth data are processed in two separate streams; in the end, the two streams are combined with a late fusion network [8].”
FPNN: Field Probing Neural Networks for 3D Data (2016, methodology): “5D data can be represented as multiple channel images, and processed by 2D CNNs [26, 10, 8].”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “neural network fuses the separate color and depth streams from images [47], adding synthetic noise to synthetic training images [11], and using 3D convolutions on an occupancy grid representation of point clouds [38].”
Deep learning in robotics: a review of recent research (2017, background): “Recent works are even starting to combine signals from multiple modalities and combine them together for unified recognition and understanding [64].”
Deep Learning Advances in Computer Vision with 3D Data (2017, neutral): “The task of RGB-D object recognition was also addressed by Eitel et al. [2015].”
Deep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not (2016, methodology): “Multi-modal combination by taking different raw sensor information (RGB, RGB-D, Lidar, etc. ) as input (Eitel et al., 2015) is still deserved to explore.”
Multispectral Pedestrian Detection using Deep Fusion Convolutional Neural Networks (2016, methodology): “Similar late-fusion architectures have been used in [10] to perform RGB-D object recognition and in [11] for action recognition in videos.”
Modality Distillation with Multiple Stream Networks for Action Recognition (2018, methodology): “The masked depth maps are converted to a three channel map via a jet mapping, as in [16].”
LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks (2018, methodology): “Eitel et al. [13] proposed to carry out objection recognition by fusing depth maps and color images with a CNN.”
Deep Projective 3D Semantic Segmentation (2017, background): “Previous work [3] has shown that a 3-channel jet colormap representation of the depth image better beneﬁts from pre-training on RGB datasets, such as ImageNet.”
A General Pipeline for 3D Detection of Vehicles (2018, background): “Fusion Approaches: In [19], the sparse point cloud is converted to a dense depth image, which is similar to a stereo one.”
Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks (2017, neutral): “Relevant architectures include two parallel networks CNN merging in the same fully connected layers [21] (for RGB-D data classification) and two CNN streams merging in the middle [22] (for fingertip detection).”
High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference (2017, background): “These depth images are enhanced with jet color mapping [10].”
Joint Human Detection and Head Pose Estimation via Multistream Networks for RGB-D Videos (2017, methodology): “[50], plane-view projection [36], two-stream CNN networks [42], fast R-CNN [33], and two other classic approaches: the depth-based detector proposed by the comboHOD detector [8] and the color-depth detector proposed by Choi et al.”
DeepFruits: A Fruit Detection System Using Deep Neural Networks (2016, background): “We can also observe that the number of parameters of a network is a critical component to increase detection performance, as mentioned by [25,30].”
AdapNet: Adaptive semantic segmentation in adverse environmental conditions (2017, methodology): “Over the past years, Deep Convolutional Neural Network (DCNN) approaches have achieved impressive results in various visual perception problems including object classification [8], [21], [22], detection [6], [18] and scene parsing [1], [15], [17].”