Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection

By Hao Chen, Youfu Li, Dan Su. Pattern Recognition, 2019

336 citations recorded by the source corpus.

Abstract Paired RGB and depth images are becoming popular multi-modal data adopted in computer vision tasks. Traditional methods based on Convolutional Neural Networks (CNNs) typically fuse RGB and depth by combining their deep representations in a late stage with only one path, which can be ambiguous and insufficient for fusing large amounts of cross-modal data. To address this issue, we propose a novel multi-scale multi-path fusion network with cross-modal interactions (MMCI), in which the traditional two-stream fusion architecture with single fusion path is advanced by diversifying the fusion path to a global reasoning one and another local capturing one and meanwhile introducing cross-modal interactions in multiple layers. Compared to traditional two-stream architectures, the MMCI net is able to supply more adaptive and flexible fusion flows, thus easing the optimization and enabling sufficient and efficient fusion. Concurrently, the MMCI net is equipped with multi-scale perception ability (i.e., simultaneously global and local contextual reasoning). We take RGB-D saliency detection as an example task. Extensive experiments on three benchmark datasets show the improvement of the proposed MMCI net over other state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Sensory Systems

View Paper: https://www.semanticscholar.org/paper/fe9b777ed7d9c9c8d6c772eeac0fe5bf176995a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Sensory Systems

Citations in the corpus

Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision (2020, background): “multi-path fusion network with that follows a two-stream fusion architecture with cross-modal interactions in multiple layers for coordinated representations [213].”
Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks (2019, background): “developed a series of novel approaches for this field, such as hidden structure transfer [43], a complementarity fusion module [49], an attention-aware component [80], [82], and dilated convolutions [81].”
Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs (2019, background): “[17] proposed a multiscale multi-path fusion network with cross-modal interactions, which could explore deep connections between RGB images and depth images.”
Multilevel and Multiscale Network for Single-Image Super-Resolution (2019, background): “In order to obtain multiscale features, some multiscale based reconstruction works [15]–[18] have been proposed.”
Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, methodology): “etwork pipeline. A.2 RGB-D Salient object detection Dataset. There are ve main RGB-D SOD datasets which are NJUD [23], RGBD135 [9] NLPR [34], SSD [74] and SIP [16]. We adopt the same splitting way as [3,5,18,71,36] to guarantee a fair comparison. We split 1,485 samples from NJUD and 700 samples from”
BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network (2020, methodology): “To efficiently integrate the RGB and depth cues for SOD, researchers have explored different multi-modal strategies [3, 5], and have achieved encouraging results.”
JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection (2020, methodology): “Among the competitors, DF [49], AFNet [60], CTMF [26], MMCI [6], PCF [4], TANet [5], CPFP [77], D3Net [18], DMRA [46] are recent deep learningbased methods, while ACSD [34], LBE [20], DCMC [13], MDSF [55], SE [25] are traditional techniques using various hand-crafted features/hypotheses.”
Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection (2019, methodology): “We compare our method with 16 state-of-the-art ones, including 5 latest CNNs-based RGB-D methods: PCA [3], PDNet [48], MMCI [4], CTMF [18], DF [35]; 5 traditional RGB-D methods: MB [49], CDCP [50], NLPR [32], DES [8], DCMC [9]; 6 top ranking CNNs-based RGB methods: PiCANet [30], PAGRN [47], R(3)Net ”