Learning to predict where humans look

By Tilke Judd, Krista A. Ehinger, Frédo Durand, Antonio Torralba. IEEE International Conference on Computer Vision, 2009

2,189 citations recorded by the source corpus.

For many applications in graphics, design, and human computer interaction, it is essential to understand where humans look in a scene. Where eye tracking devices are not a viable option, models of saliency can be used to predict fixation locations. Most saliency approaches are based on bottom-up computation that does not consider top-down image semantics and often does not match actual eye movements. To address this problem, we collected eye tracking data of 15 viewers on 1003 images and use this database as training and testing examples to learn a model of saliency based on low, middle and high-level image features. This large database of eye tracking data is publicly available with this paper.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/c7614898de1cfee1a3a1c564f52c1e67879c29b3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Differential Attention for Visual Question Answering (2018, methodology): “However, as noted by [3], using a saliency based method [10] that is trained on eye tracking data to obtain a measure of where people look in a task independent manner results in more correlation with human attention (0.”
Efficient, simultaneous detection of multi-class geospatial targets based on visual saliency modeling and discriminative learning of sparse coding (2014, methodology): “These models (Achanta et al., 2008, 2009; Borji, 2012; Han et al., 2006, 2011; Harel et al., 2006; Hou and Zhang, 2007; Itti et al., 1998; Judd et al., 2009; Zhang et al., 2008) generally assume that locations in the visual field that are distinctive from their contextual background are more likely ”
A Benchmark of Computational Models of Saliency to Predict Human Fixations (2012, methodology): “To facilitate comparisons, several authors have produced openly available databases of images with fixations from eye tracking experiments [41] [42] [10] [6] [37] [61] and executable code of models that others can use.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, background): “• AUC-Judd (AUC-J): AUC-J [38] is a variant of AUC proposed by Judd et al.”
Faster gaze prediction with dense networks and Fisher pruning (2018, methodology): “As in [20], the model was ﬁrst pre-trained using the SALICON dataset [13] while using the MIT1003 dataset [15] for validation.”
Methods for comparing scanpaths and saliency maps: strengths and weaknesses (2012, methodology): “This dispersion (also called inter observer congruency [IOC 1 ]) is contingent upon a number of factors.”
Training Object Class Detectors from Eye Tracking Data (2014, background): “However, fixation data only provides a rough indication of the spatial extent of an object: humans have a tendency to fixate the center of an object [33], and within animate objects (humans and animals), there is a strong preference for fixating faces [24].”
Geodesic Saliency Using Background Priors (2012, background): “Early works [1, 2] in this field are mostly based on biologically inspired models (where a human looks) and is evaluated on human eye fixation data [3, 4].”
Saliency Detection via Graph-Based Manifold Ranking (2013, background): “We note that saliency models have been developed for eye fixation prediction [6, 14, 15, 17, 19, 25, 33] and salient object detection [1, 2, 7, 9, 23, 24, 32].”
Dual Attention Guided Gaze Target Detection in the Wild (2021, background): “[20] to assess the confidence of the predicted heatmap.”
First Person Action-Object Detection with EgoNet (2016, methodology): “We include the following baseline methods in our comparisons: (1-2) GBVS [19] and Judd [23]: two bottom-up visual saliency methods; (3) FP-MCG [1]: a multiscale object segmentation proposal method that was trained on our ﬁrst-person dataset; (4) Handled+Viewed Object: our trained method that detects”
Salient Region Detection by UFO: Uniqueness, Focusness and Objectness (2013, background): “[18] integrate high-level information, making their methods potentially suitable for specific tasks.”
Hierarchical Image Saliency Detection on Extended CSSD (2016, result): “But the results in (b), produced by a previous local method, only highlight a few edges that scatter in the image.”
U-CAM: Visual Explanation Using Uncertainty Based Class Activation Maps (2019, methodology): “However, using a saliency-based method [20] that is trained on eye-tracking data to obtain a measure of where people look in a task-independent manner, results in more correlation with human attention (0.”
Understanding and Predicting Image Memorability at a Large Scale (2015, methodology): “Saliency: Using images from the Fixation Flickr [14] dataset, we explore the relationship between human fixations and memorability.”
Predicting human gaze beyond pixels. (2014, methodology): “, 2009) and MIT (Judd et al., 2009) models that include”
State-of-the-Art in Visual Attention Modeling (2013, methodology): “Because of this, a trivial Gaussian blob model scores higher than almost all saliency models (see [166]).”