A deep multi-level network for saliency prediction

By Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara. International Conference on Pattern Recognition, 2016

375 citations recorded by the source corpus.

This paper presents a novel deep architecture for saliency prediction. Current state of the art models for saliency prediction employ Fully Convolutional networks that perform a non-linear combination of features extracted from the last convolutional layer to predict saliency maps. We propose an architecture which, instead, combines features extracted at different levels of a Convolutional Neural Network (CNN). Our model is composed of three main blocks: a feature extraction CNN, a feature encoding network, that weights low and high level feature maps, and a prior learning network. We compare our solution with state of the art saliency models on two public benchmarks datasets. Results show that our model outperforms under all evaluation metrics on the SALICON dataset, which is currently the largest public dataset for saliency prediction, and achieves competitive results on the MIT300 benchmark. Code is available at https://github.com/marcellacornia/mlnet.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Sensory Systems

View Paper: https://www.semanticscholar.org/paper/57e143c96a41ba81d9d59120a1cd0dc04905d3f1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Sensory Systems

Citations in the corpus

Deep Visual Attention Prediction (2017, background): “2(c), which is adopted in [42], [43], [49].”
Object Detection With Deep Learning: A Review (2018, background): “proposed an architecture which combines features extracted at different levels of the CNN [152].”
Gaze Prediction in Dynamic 360° Immersive Videos (2018, background): “Some work also attempts to use low-level appearance and motion cues as inputs or extend deep learning approaches to more complex scenarios such as stereo images [9] or videos [4] [7] [11] [31] [35] [32] [16] [12] [6] [18].”
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (2017, methodology): “Cornia et al. [5] proposed an architecture that combines features extracted at different levels of a DCNN.”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, methodology): “• INTEXP(SAL) uses RGB plus heatmaps from a pretrained saliency model [13] as input, which highlight salient objects but are devoid of affordance cues.”
Predicting the Driver's Focus of Attention: The DR(eye)VE Project (2017, methodology): “Moreover, MLNet low performance testiﬁes for the need of accounting the temporal correlation between consecutive frames that distinguishes the image and video saliency tasks.”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, background): “Saliency prediction has been extensively studied by the computer vision community and, in the last few years, has achieved a considerable improvement thanks to the large spread of deep neural networks [7, 8, 16, 18, 26, 27, 35].”
Saliency in VR: How Do People Explore Virtual Environments? (2016, methodology): “For each projection, we compute a saliency map using the state-of-the-art MLNet saliency predictor [10], and then optionally multiply it by the latitudinal equator bias we derived in Section 4.”
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, methodology): “Accordingly, we proposed a new architecture, called ML-Net [15], which improved previous attempts by using features coming from multiple layers of a CNN and adding a learned prior map.”
Grounded Human-Object Interaction Hotspots From Video (2018, background): “The SALIENCY baselines capture a generic notion of spatial importance.”
Deep Facial Expression Recognition: A Survey (2018, methodology): “n changes, and cropped these regions as the input of DSAE. Other researches proposed to automatically learn the key parts for facial expression. For example, [159] employed a deep multi-layer network [160] to detect the saliency map which put intensities on parts demanding visual attention. And [161”