Cross-modal adaptation for RGB-D detection

By Judy Hoffman, Saurabh Gupta, Jian Leong, Sergio Guadarrama, Trevor Darrell. 2016

91 citations recorded by the source corpus.

In this paper we propose a technique to adapt convolutional neural network (CNN) based object detectors trained on RGB images to effectively leverage depth images at test time to boost detection performance. Given labeled depth images for a handful of categories we adapt an RGB object detector for a new category such that it can now use depth images in addition to RGB images at test time to produce more accurate detections. Our approach is built upon the observation that lower layers of a CNN are largely task and category agnostic and domain specific while higher layers are largely task and category specific while being domain agnostic. We operationalize this observation by proposing a mid-level fusion of RGB and depth CNNs. Experimental evaluation on the challenging NYUD2 dataset shows that our proposed adaptation technique results in an average 21% relative improvement in detection performance over an RGB-only baseline even when no depth training data is available for the particular category evaluated. We believe our proposed technique will extend advances made in computer vision to RGB-D data leading to improvements in performance at little additional annotation effort.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Cross Modal Distillation for Supervision Transfer (2015, background): “This can naturally be extended to incorporate annotations from auxiliary categories as explored in [21], but we defer this to future work.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[87] learn a depth network by transferring the knowledge from RGB network and fuse the information across modalities, which improves the object recognition performance at test time.”
Label Efficient Learning of Transferable Representations acrosss Domains and Tasks (2017, result): “Weight sharing of CNN parameters [58, 22, 21, 3] and minimizing the distribution discrepancy of network activations [51, 65, 30] have also shown convincing results.”
Deep Visual Domain Adaptation: A Survey (2018, methodology): “Most work commonly uses the negative loglikelihood of the ground truth class with softmax as their training loss, L = − ∑N i=0 yi log ŷi (ŷi are the softmax predictions of the model, which represent class probabilities) [95], [40], [34], [70], [45], [100].”
CNNs-Based RGB-D Saliency Detection via Cross-View Transfer and Multiview Fusion (2018, result): “This inferiority of early fusion strategy demonstrates that higher layers in CNNs with task-specific and domain-agnostic high-level features are more suitable for cross-view fusion, which is consistent with the observation in many other tasks [43], [48].”
KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving (2018, background): “in the challenging scenarios to RGB images ( [13], [21]–[23]).”
SemanticFusion: Dense 3D semantic mapping with convolutional neural networks (2016, background): “Some recent work has explored other ways to incorporate depth information [8], but such an approach requires duplication of the lower network parameters and was infeasible in our system due to GPU memory limitations.”
Adapting Models to Signal Degradation using Distillation (2016, methodology): “Recent works have used a similar idea to ours to learn representations of depth from color using RGB-D data [21, 23], representations of video from ambient sound [37] and vice-versa [1], as well as visual representations through self-supervised colorization [29, 54].”