CNNs-Based RGB-D Saliency Detection via Cross-View Transfer and Multiview Fusion

By Junwei Han, Haorong Chen, Nian Liu, Chenggang Clarence Yan, Xuelong Li. IEEE Transactions on Cybernetics, 2018

378 citations recorded by the source corpus.

Salient object detection from RGB-D images aims to utilize both the depth view and RGB view to automatically localize objects of human interest in the scene. Although a few earlier efforts have been devoted to the study of this paper in recent years, two major challenges still remain: 1) how to leverage the depth view effectively to model the depth-induced saliency and 2) how to implement an optimal combination of the RGB view and depth view, which can make full use of complementary information among them. To address these two challenges, this paper proposes a novel framework based on convolutional neural networks (CNNs), which transfers the structure of the RGB-based deep neural network to be applicable for depth view and fuses the deep representations of both views automatically to obtain the final saliency map. In the proposed framework, the first challenge is modeled as a cross-view transfer problem and addressed by using the task-relevant initialization and adding deep supervision in hidden layer. The second challenge is addressed by a multiview CNN fusion model through a combination layer connecting the representation layers of RGB view and depth view. Comprehensive experiments on four benchmark datasets demonstrate the significant and consistent improvements of the proposed approach over other state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/a6290d805363bbf0c2a21c60cd487001a05cabdb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection (2020, methodology): “Generally, as summarized by previous literature [4, 77], the above approaches can be divided into three categories: (a) Early-fusion [18,31,40,55], (b) late-fusion [26,60] and (c) middle-fusion [4–6 , 80 ].”
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders (2020, methodology): “[41] [36] [8] [44] [66] [28] [20] [10] [51] [22] [43] [54] [24] [7] [5] [6] [64] [61] Ours”
Advanced Multi-Sensor Optical Remote Sensing for Urban Land Use and Land Cover Classification: Outcome of the 2018 IEEE GRSS Data Fusion Contest (2019, background): “Following the great success of deep learning in computer vision field [19]–[21], many deep models have been proposed to address the remote sensing image classification task [22]–[28].”
Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection (2019, methodology): “[25] proposed a two-view(RGB and depth) CNN to obtain the feature from RGB images and corresponding depth image, then simultaneously connected these feature with a new fully connected layer to get the ﬁnal saliency map.”
BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network (2020, background): “es to be manually designed as input, and thus cannot be trained in an end-to-end manner. More recently, researchers have begun to extract deep RGB and depth features using CNNs in a bottom-up fashion [69]. Unlike handcrafted features, deep features contain a lot of contextual and semantic informatio”
Visual Saliency Transformer (2021, methodology): “Although some methods utilized fully connected layers [36, 22], global pooling layers [44, 37, 65], and non-local modules [38, 7] to incorporate the global context, they only did such in certain layers and the standard CNN-based architecture remains unchanged.”
Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection (2019, methodology): “We compare our method with 16 state-of-the-art ones, including 5 latest CNNs-based RGB-D methods: PCA [3], PDNet [48], MMCI [4], CTMF [18], DF [35]; 5 traditional RGB-D methods: MB [49], CDCP [50], NLPR [32], DES [8], DCMC [9]; 6 top ranking CNNs-based RGB methods: PiCANet [30], PAGRN [47], R(3)Net ”
Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks (2019, background): “0.4 0.6 P r ec i s i on SE [42] DF [44] AFNet [61] CTMF [43] MMCI [81] PCF [49] TANet [82] CPFP [51] D3Net LHM [39] CDB [47] DESM [38] GP [40] CDCP [46] ACSD [37] LBE [41] DCMC [55] MDSF [45] Figure 9.”
Salient Object Detection via Integrity Learning (2021, background): “[59] built an early deep model to transfer the structure of an RGB-based deep neural network to process depth view data and fuse the deep representations of both views to generate the final saliency prediction results.”
Review of Visual Saliency Detection With Comprehensive Information (2018, background): “[61] firstly exposed the cross-modal discrepancy in the RGBD data and proposed two cross-modal transfer”
Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, methodology): “art Results. The performance of the proposed model is compared with ten state-of-the-art approaches on ve benchmark 16 Zhao et al. datasets, including the DES [9], DCMC [11], CDCP [75], DF [38], CTMF [18], PCA [3], MMCI [5], TANet [4], CPFP [71] and DMRA [36]. For fair comparisons, all the saliency ”
Progressively Complementarity-Aware Fusion Network for RGB-D Salient Object Detection (2018, methodology): “However, benefit from more sufficient fusion of multi-modal and multi-level features, our method still achieves much better performance than the DF and CTMF models.”