Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

By Deng-Ping Fan, Zheng Lin, Jiaxing Zhao, Yun Liu, Zhao Zhang, Qibin Hou, Menglong Zhu, Ming-Ming Cheng. IEEE Transactions on Neural Networks and Learning Systems, 2019

640 citations recorded by the source corpus.

The use of RGB-D information for salient object detection (SOD) has been extensively explored in recent years. However, relatively few efforts have been put toward modeling SOD in real-world human activity scenes with RGB-D. In this article, we fill the gap by making the following contributions to RGB-D SOD: 1) we carefully collect a new Salient Person (SIP) data set that consists of ~1 K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and background s; 2) we conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research, and we systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven data sets containing a total of about 97k images; and 3) we propose a simple general architecture, called deep depth-depurator network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning, respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background-changing application with a speed of 65 frames/s on a single GPU. All the saliency maps, our new SIP data set, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/8cdb3279f8b29b6a19c228af02e8ee1deb591e21

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection (2022, methodology): “the same training dataset as in [78] and [80], which consists of”
Camouflaged Object Detection (2020, background): “(a) Image (b) Generic object (c) Salient object (d) Camouflaged object Figure 2: Given an input image (a), we present the ground-truth for (b) panoptic segmentation [30] (which detects generic objects [39,44] including stuff and things), (c) salient instance/object detection [16, 33, 61, 76] (which ”
JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection (2020, methodology): “Generally, as summarized by previous literature [4, 77], the above approaches can be divided into three categories: (a) Early-fusion [18,31,40,55], (b) late-fusion [26,60] and (c) middle-fusion [4–6 , 80 ].”
Structure-Measure: A New Way to Evaluate Foreground Maps (2017, background): “Finally, we also evaluate our measure over SOD dataset, which is a subset of the BSDS dataset.”
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders (2020, methodology): “Saliency detection is subjective to some extent, and it is common to have multiple annotators to label one image, and the final ground truth saliency map is obtained through majority voting strategy [18].”
EGNet: Edge Guidance Network for Salient Object Detection (2019, background): “Besides, there are many works focusing on video salient object detection [12, 54] and RGB-D salient object detection [11, 66].”
RGB-D salient object detection: A survey (2020, methodology): “, D3Net [34], SSF [167], A2dele [106], S2MA [82], ICNet [71], JL-DCF [42], and UC-Net [160]).”
Stacked Cross Refinement Network for Edge-Aware Salient Object Detection (2019, background): “Salient object detection [1, 5, 10, 11, 17] aims to detect and segment the most attractive objects in images or videos.”
Visual Saliency Transformer (2021, methodology): “For RGB-D SOD, we use nine widely used benchmark datasets: STERE [46] (1,000 image pairs), LFSD [33] (100 image pairs), RGBD135 [9] (135 image pairs), SSD [90] (80 image pairs), NJUD [26] (1,985 image pairs), NLPR [51] (1,000 image pairs), DUTLF-Depth [52] (1,200 image pairs), SIP [15] (929 image pa”
BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network (2020, background): “Strategies NJU2K [34] NLPR [51] STERE [49] DES [9] LFSD [41] SSD [77] SIP [20] Sα ↑ M ↓ Sα ↑ M ↓ Sα ↑ M ↓ Sα ↑ M ↓ Sα ↑ M ↓ Sα ↑ M ↓ Sα ↑ M ↓”