DeepGaze II: Reading fixations from deep features trained on object recognition

By Matthias Kümmerer, Thomas S. A. Wallis, Matthias Bethge. arXiv.org, 2016

317 citations recorded by the source corpus.

Here we present DeepGaze II, a model that predicts where people look in images. The model uses the features from the VGG-19 deep neural network trained to identify objects in images. Contrary to other saliency models that use deep features, here we use the VGG features for saliency prediction with no additional fine-tuning (rather, a few readout layers are trained on top of the VGG features to predict saliency). The model is therefore a strong test of transfer learning. After conservative cross-validation, DeepGaze II explains about 87% of the explainable information gain in the patterns of fixations and achieves top performance in area under the curve metrics on the MIT300 hold-out benchmark. These results corroborate the finding from DeepGaze I (which explained 56% of the explainable information gain), that deep features trained on object recognition provide a versatile feature space for performing related visual tasks. We explore the factors that contribute to this success and present several informative image examples. A web service is available to compute model predictions at this http URL.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d160d375a7c187dfe10c110e9e733f36accb87e6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency (2021, background): “Twitter’s saliency model builds on the deep learning architecture DeepGaze II [51], which leverages feature maps pretrained on object recognition.”
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, methodology): “For this reason, hazy or blurred saliency maps like the ones predicted by [24] achieve high AUC values [69], [34], despite being visually very different from the groundtruth annotations, as we will show in the following.”
Grounded Human-Object Interaction Hotspots From Video (2018, background): “The SALIENCY baselines capture a generic notion of spatial importance.”
Deep Visual Attention Prediction (2017, methodology): “A updated version of DeepGaze [42], called DeepGaze II [43], recently was proposed that provids a deeper network with VGG-19 [22], where the attention information is directly inferred from the original VGGNet, without ﬁne-tuning on attention dataset.”
Generalisation in humans and deep neural networks (2018, background): “1 Secondly, features learned on one task often transfer to only loosely related tasks, such as from classification to saliency prediction [33], emotion recognition [34], medical imaging [35] and a large number of other transfer learning tasks [36].”
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (2017, methodology): “A new version called DeepGaze 2 [21] using VGG-19 [27] and trained with the SALICON dataset [12] has recently achieved the best results in the MIT saliency benchmark.”
Faster gaze prediction with dense networks and Fisher pruning (2018, methodology): “In particular, reusing image representations trained for the task of object recognition have proven to be very useful [19, 20].”
What are the Visual Features Underlying Human Versus Machine Vision? (2017, methodology): “Because theory holds that attention is driven by both bottom-up and top-down mechanisms [10], we compared importance maps to saliency maps derived from models for both kinds of attention, referred to hereafter as bottomup saliency [10]3 and top-down attention derived from the DeepGaze II model [15].”
Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency (2019, methodology): “reused AlexNet [46] and VGG-19 [47] to generate high-dimensional features for fixation prediction in [48] and [49], respectively.”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, background): “[27, 28] based their models on two popular convolutional networks (AlexNet [25] and VGG19 [41]) obtaining adequate results, despite the network parameters were not ne-tuned on a saliency dataset.”
Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “Typical saliency networks are convolutional neural networks [Jetley et al. , 2016; K¨ummerer et al. , 2016; Kruthiventi et al. , 2017] or convolutional long short-term memory (LSTM) networks [Cornia et al. , 2018b].”