DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations

By Srinivas S. S. Kruthiventi, Kumar Ayush, R. Venkatesh Babu. IEEE Transactions on Image Processing, 2015

512 citations recorded by the source corpus.

Understanding and predicting the human visual attention mechanism is an active area of research in the fields of neuroscience and computer vision. In this paper, we propose DeepFix, a fully convolutional neural network, which models the bottom–up mechanism of visual attention via saliency prediction. Unlike classical works, which characterize the saliency map using various hand-crafted features, our model automatically learns features in a hierarchical fashion and predicts the saliency map in an end-to-end manner. DeepFix is designed to capture semantics at multiple scales while taking global context into account, by using network layers with very large receptive fields. Generally, fully convolutional nets are spatially invariant—this prevents them from modeling location-dependent patterns (e.g., centre-bias). Our network handles this by incorporating a novel location-biased convolutional layer. We evaluate our model on multiple challenging saliency data sets and show that it achieves the state-of-the-art results.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1295cbaf3b03de2eb8c79530289f5939d7819e5c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, methodology): “…paradigm which aims to compute an output as a weighted sum of different M. Cornia, L. Baraldi and R. Cucchiara are with the Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, Modena, Italy (e-mail: { marcella.cornia, lorenzo.baraldi, rita.cucchiara } @unimore.it).”
Salient Object Detection in the Deep Learning Era: An In-Depth Survey (2019, background): “• FSN [79], inspired by the observation that salient objects typically gain most of human eye-fixations [59], fuses the outputs of a fixation stream [99] and a semantic stream [95] into an inception-segmentation module to predict saliency.”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “Compared to the state-of-the-art deep model DeepFix [24] method, SACT does competitively.”
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (2017, methodology): “Solutions based on MSE aim at maximizing the peak signal-to-noise ratio (PSNR).”
Faster gaze prediction with dense networks and Fisher pruning (2018, background): “Kruthiventi et al. [18] also tried to ﬁnetune a pretrained network by us-ing a smaller learning rate for pretrained parameters than for other parameters.”
Revisiting Video Saliency Prediction in the Deep Learning Era (2021, background): “Recently, deep learning based static saliency models [28], [29], [30], [53], [54], [55], [56] have achieved great improvements, relying on the powerful end-to-end learning ability of neural networks and the availability of large-scale static saliency data-sets [2].”
DeepGaze II: Reading fixations from deep features trained on object recognition (2016, methodology): “Note that we currently do not include models that improved on DeepGaze I on the MIT benchmark (SALICON, DeepFix and PDP) in this evaluation because the code for these models is not publically available.”
Deep Visual Attention Prediction (2017, methodology): “To demonstrate the effectiveness of the proposed deep attention model in predicting eye ﬁxations, we evaluated it by comparison to 13 state-of-the-art models, including six classical models: ITTI [28], GBVS [38], Judd Model [24], BMS [57], CAS [41], AIM [56], and seven deep learning based models: De”
A deep multi-level network for saliency prediction (2016, background): “The recent publication of the large dataset SALICON [12], collected thanks to crowd-sourcing techniques, allows researches to increase the number of convolutional layers reducing the overﬁtting risk [13], [14].”
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, background): “…could have broad application throughout software engineering, such as in models to suggest readable function and class names [3], summarizing source code [5, 43], predicting bugs [60], detecting code clones [73], comment generation [42], ﬁxing syntactic errors [47], and variable de-obfuscation [9].”
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model (2018, background): “Deep learning based static saliency models [54, 35, 24, 39, 47, 29, 56, 57] have achieved astonishing improvements, relying on the powerful end-to-end learning ability of neural network and the availability of large-scale static saliency datasets [31].”
Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “Typical saliency networks are convolutional neural networks [Jetley et al., 2016; Kümmerer et al., 2016; Kruthiventi et al., 2017] or convolutional long short-term memory (LSTM) networks [Cornia et al.”
AGIL: Learning Attention from Human for Visuomotor Tasks (2018, background): “In recent years, this approach has made tremendous progress due to large benchmark datasets such as [20, 18] and deep neural networks [4, 24, 21, 41].”
Gaze Prediction in Dynamic 360° Immersive Videos (2018, background): “Recently, advances in deep learning have produced more accurate models [20] [33] [28] [37] [24].”