Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet

By Matthias Kümmerer, Lucas Theis, Matthias Bethge. 2014

446 citations recorded by the source corpus.

Recent results suggest that state-of-the-art saliency models perform far from optimal in predicting fixations. This lack in performance has been attributed to an inability to model the influence of high-level image features such as objects. Recent seminal advances in applying deep neural networks to tasks like object recognition suggests that they are able to capture this kind of structure. However, the enormous amount of training data necessary to train these networks makes them difficult to apply directly to saliency prediction. We present a novel way of reusing existing neural networks that have been pretrained on the task of object recognition in models of fixation prediction. Using the well-known network of Krizhevsky et al. (2012), we come up with a new saliency model that significantly outperforms all state-of-the-art models on the MIT Saliency Benchmark. We show that the structure of this network allows new insights in the psychophysics of fixation selection and potentially their neural implementation. To train our network, we build on recent work on the modeling of saliency as point processes.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, methodology): “ion. The CNN models instantiated in [276] are however not deep enough because the maximum number of layers is capped at three. By using a pre-trained and deeper CNN model with 5 convolutional layers, [277] (Deep Gaze) learns a separate saliency model to jointly combine the responses from every CNN l”
Faster gaze prediction with dense networks and Fisher pruning (2018, methodology): “In particular, reusing image representations trained for the task of object recognition have proven to be very useful [19, 20].”
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention (2020, background): “Predicting what people visually attend to in images (saliency prediction) is a long-standing challenge in neuroscience and computer vision [4, 6, 40].”
Predicting the Driver's Focus of Attention: The DR(eye)VE Project (2017, background): “the-art results on public benchmarks [15], [16], [28], [34], [37].”
Image Style Transfer Using Convolutional Neural Networks (2016, background): “In light of the striking similarities between performanceoptimised artificial neural networks and biological vision [11, 31, 3, 19, 16], it is thus tempting to speculate that the human ability to abstract content from style – and therefore our ability to create and enjoy art – might also be primaril”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, background): “Saliency prediction has been extensively studied by the computer vision community and, in the last few years, has achieved a considerable improvement thanks to the large spread of deep neural networks [7, 8, 16, 18, 26, 27, 35].”
DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection (2016, background): “, eye fixation prediction [32, 33] and salient object detection [34-36].”
Deep Neural Networks as Scientific Models. (2019, methodology): “Another example involves recent ﬁ ndings on transfer learning, where a DNN trained on one task (usuallyobjectcategorisation) is evaluated in another task [74 – 78].”
TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking (2015, background): “[26, 23, 18] demonstrated that eye tracking datasets on a larger scale are critical to leverage the power of machine learning techniques and greatly improve the performance of visual attention models.”
Recurrent Mixture Density Network for Spatiotemporal Visual Attention (2016, background): “Recent work (Mauthner et al., 2015; Hossein Khatoonabadi et al., 2015; Mathe & Sminchisescu, 2015; Stefan Mathe, 2013; Kümmerer et al., 2015; Rudoy et al., 2013) has shown that it may be possible to accurately reproduce gazing patterns of human subjects attending to images and videos.”
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (2017, methodology): “DeepGaze [19] provided a deeper network using the well-know AlexNet [16], with pre-trained weights on Imagenet [6] and with a readout network on top whose inputs consisted of some layer outputs of AlexNet.”