Shallow and Deep Convolutional Networks for Saliency Prediction

By Junting Pan, Elisa Sayrol, Xavier Giró-i-Nieto, Kevin McGuinness, Noel Edward O'Connor. Computer Vision and Pattern Recognition, 2016

497 citations recorded by the source corpus.

The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. Two designs are proposed: a shallow convnet trained from scratch, and a another deeper solution whose first three layers are adapted from another network trained for classification. To the authors' knowledge, these are the first end-to-end CNNs trained and tested for the purpose of saliency prediction.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9528e2e8c20517ab916f803c0371abb4f0ed488b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping (2019, methodology): “We test eight state-of-the-art attention models including ITTI [17], AIM [18], GBVS [19], SUN [21], DVA [22], SIG [68], CAS [69] and SalNet [34].”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “[40] presents two classes of networks, one shallow and one deep, that achieve similar prediction error for saliency prediction.”
Faster gaze prediction with dense networks and Fisher pruning (2018, background): “Similarly, Pan et al. [26] trained networks end-to-end while initializing only a few layers with parameters obtained from pretrained networks but have since been outperformed by DeepGaze II and other recent approaches.”
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency (2019, methodology): “3) The Deep Learning Group (denoted as D Group) contains five deep learning-based models, including eDN [32], iSEEL [52], DVA [8], SalNet [7] and STS [9].”
A deep multi-level network for saliency prediction (2016, methodology): “The recent publication of the large dataset SALICON [12], collected thanks to crowd-sourcing techniques, allows researches to increase the number of convolutional layers reducing the overﬁtting risk [13], [14].”
Interactive Image Segmentation via Backpropagating Refinement Scheme (2019, background): “Thanks to the release of large image datasets [23] and the use of convolution layers, deep-learning-based algorithms have been showing remarkable performances in segmentation problems: semantic segmentation [13, 30, 35, 6], saliency detection [29, 36], and object proposal [39, 38].”
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model (2018, methodology): “OM-CNN, Two-stream, SALICON, DVA, Shallow-Net, and Deep-Net are deep learning models, and others are classical saliency models.”
Gaze Prediction in Dynamic 360° Immersive Videos (2018, methodology): “Similar to FOV saliency, we also feed the panorama image into SalNet for spatial saliency detection, and use the optical flow calculated with the panorama images as a temporal saliency estimation.”
Revisiting Video Saliency Prediction in the Deep Learning Era (2021, methodology): “For the sake of completeness, we further include seven state-of-the-art static attention models: ITTI [26], GBVS [27], SALICON [28], SalGAN [31], DVA [29], Shallow-Net [30], and Deep-Net [30].”
Deep Level Sets for Salient Object Detection (2017, methodology): “However, when applying CNNs to pixel-wise saliency labeling, it may suffer from some limitations.”
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, methodology): “Image DeepGazeII [24] SalNet [40] ML-Net [15] SAM-VGG SAM-ResNet GT Image eDN [25] MrCNN [37] ML-Net [15] SAM-VGG SAM-ResNet GT”
Object Detection With Deep Learning: A Review (2018, background): “formulated saliency prediction as a minimization optimization on the Euclidean distance between the predicted saliency map and the ground truth and proposed two kinds of architectures [147]: a shallow one trained from scratch and a deeper one”
Deep Visual Attention Prediction (2017, methodology): “In [15], two deep learning models with shallow and deep network architectures were exploited for saliency prediction.”
CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes (2018, methodology): “The recent development of congested scene analysis relays on DNN-based methods because of the high accuracy they have achieved in semantic segmentation tasks [7, 8, 9, 10, 11] and the significant progress they have made in visual saliency [12].”
Learning Unsupervised Video Object Segmentation Through Visual Attention (2019, methodology): “Recently, deep learning based attention models [71, 28, 55, 73, 75] were proposed and generally yield better performance.”
Non-local Deep Features for Salient Object Detection (2017, methodology): “While some methods apply a straight forward convolutional neural net (CNN) model [36], others have proposed a model tailored to the saliency detection problem [25, 26, 29, 43, 50].”