Visual saliency based on multiscale deep features

By Guanbin Li, Yizhou Yu. 2015

1,346 citations recorded by the source corpus.

Visual saliency is a fundamental problem in both cognitive and computational sciences, including computer vision. In this paper, we discover that a high-quality visual saliency model can be learned from multiscale features extracted using deep convolutional neural networks (CNNs), which have had many successes in visual recognition tasks. For learning such saliency models, we introduce a neural network architecture, which has fully connected layers on top of CNNs responsible for feature extraction at three different scales. We then propose a refinement method to enhance the spatial coherence of our saliency results. Finally, aggregating multiple saliency maps computed for different levels of image segmentation can further boost the performance, yielding saliency maps better than those generated from a single segmentation. To promote further research and evaluation of visual saliency models, we also construct a new large database of 4447 challenging images and their pixelwise saliency annotations. Experimental results demonstrate that our proposed method is capable of achieving state-of-the-art performance on all public benchmarks, improving the F-Measure by 5.0% and 13.2% respectively on the MSRA-B dataset and our new dataset (HKU-IS), and lowering the mean absolute error by 5.7% and 35.1% respectively on these two datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, methodology): “Similar to [167], [168], the CNN model used in”
Res2Net: A New Multi-Scale Backbone Architecture (2019, methodology): “Following [18], we train those two models using the MSRA-B dataset [27], and evaluate results on ECSSD [45], PASCAL-S [24], HKU-IS [23], and DUT-OMRON [46] datasets.”
Attentive Feedback Network for Boundary-Aware Salient Object Detection (2019, methodology): “The proposed model can run at a real-time speed of 26 FPS and achieves state-of-the-art performance on five large-scale salient object detection datasets including ECSSD [31], PASCAL-S [19], DUT-OMRON [32], HKU-IS [15], and test set from DUTS [27].”
Deep Level Sets for Salient Object Detection (2017, methodology): “Qualitative comparison with FT [1], HC [10], GMR [56], BSCA [47], MTDS [31], MDF [29], MCDL [62], ELD [26], LEGS [50].”
A Survey on Deep Learning Technique for Video Segmentation (2021, neutral): “GCNet [161] ECCV Memory Network Matching-based MSRA10K [84] + ECSSD [118] + HKUIS [146] + DAVIS17 [81]/YouTube-VOS [95] KMN [162] ECCV Memory Network Matching-based PASCAL VOC 2012 [103] + COCO [123] + ECSSD [118] + DAVIS17 [81]/YouTubeVOS [95] CFBI [163] ECCV FCN Matching-based COCO [123] + DAVIS17”
Non-local Deep Features for Salient Object Detection (2017, methodology): “Saliency maps produced by the GS [46], MR [49], wCtr* [52], BSCA [38], LEGS [43], MC [50], MDF [25] and DCL [26] methods compared to our NLDF method.”
Salient Object Detection in the Deep Learning Era: An In-Depth Survey (2019, methodology): “We construct a hybrid benchmark consists of 1,800 distinctive images randomly selected from 6 datasets (300 each), namely SOD [118], ECSSD [34], DUT-OMRON [52], PASCAL-S [59], HKU-IS [26] and the test set of DUTS [73].”
Visual Saliency Detection Based on Multiscale Deep CNN Features (2016, methodology): “Readers can refer to an earlier version of our paper [1] for performance comparisons on these two datasets.”
Salient Object Detection: A Discriminative Regional Feature Integration Approach (2013, methodology): “Multiscale deep features (MDF) (Li and Yu 2015) follows our discriminative feature integration framework and extracts the deep feature for the superpixels from a deep neural network that is learnt using regions from a set of labeled saliency maps.”
Global Context-Aware Progressive Aggregation Network for Salient Object Detection (2020, background): “• HKU-IS (Li and Yu 2015) includes 4,447 images and most of them have low contrast or more than one salient object;”
Dense and Low-Rank Gaussian CRFs Using Deep Embeddings (2017, methodology): “Our baselines for the saliency estimation task include (a) the Local Estimation and Global Search (LEGS) framework [36], (b) the multi-context network [39], (c) the multiscale deep features network [21], (d) the deep contrast learning networks [22] which proposes a network structure that better expl”
A Stagewise Refinement Model for Detecting Salient Objects in Images (2017, methodology): “* ECSSD [52] DUT-OMRON [53] THUR15K [11] SED [3] HKU-IS [28] DUTS [47] F-measure MAE F-measure MAE F-measure MAE F-measure MAE F-measure MAE F-measure MAE Ours 0.”
A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation (2017, methodology): “This network design of Hydra CNN is inspired by the work of Li et al. [53].”
Saliency Detection via a Multiple Self-Weighted Graph-Based Manifold Ranking (2020, methodology): “Wang et al. [29] and Li and Yu [30] used deep features extracted from convolutional neural networks, and respectively trained a ranking SVM and a classiﬁcation network to predict region saliency.”
CNNs-Based RGB-D Saliency Detection via Cross-View Transfer and Multiview Fusion (2018, background): “[30] and Li and Yu [33] fed multiscale image inputs into CNNs to incorporate multiscale features or contexts to detect saliency.”
Enhanced-alignment Measure for Binary Foreground Map Evaluation (2018, methodology): “Images in PAS-CAL dataset have more structural objects than the other 3 datasets (ECSSD, SOD, HKU-IS).”
Detect Globally, Refine Locally: A Novel Approach to Saliency Detection (2018, methodology): “We evaluate the proposed framework on five popular datasets: ECSSD [31], DUTOMRON [32], THUR15K [5], HKU-IS [17], and DUTS [27].”
BASNet: Boundary-Aware Salient Object Detection (2019, methodology): “Patch-wise Deep Methods: Encouraged by the advancement on image classification of Deep CNNs [28, 59], early deep salient object detection methods search for salient objects by classifying image pixels or super pixels into salient or non-salient classes based on the local image patches extracted from”
Uncertainty-aware Joint Salient Object and Camouflaged Object Detection (2021, methodology): “Dataset: For salient object detection, we train our model using the augmented DUTS training dataset [44], and testing on six other testing dataset, including the DUTS testing dataset, ECSSD [49], DUT [50], HKU-IS [27], THUR [4] and SOC testing dataset [8].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “MF SF [80] 0.493 GC [18] 0.539 DRFI [45] 0.690 PISA [101] 0.660 BSCA [83] 0.666 LEGS [102] 0.752 MC [112] 0.740 MDF [57] 0.764 FCN [58] 0.793 DCL [58] 0.815 DCL + CRF [58] Surface Normal Estimation: For this task surface normals are typically estimated from point cloud data, rather than directly…”