Modeling Search for People in 900 Scenes: A combined source model of eye guidance

By Krista A. Ehinger, Barbara Hidalgo-Sotelo, Antonio Torralba, Aude Oliva. 2009

330 citations recorded by the source corpus.

How predictable are human eye movements during search in real world scenes? We recorded 14 observers’ eye movements as they performed a search task (person detection) in 912 outdoor scenes. Observers were highly consistent in the regions fixated during search, even when the target was absent from the scene. These eye movements were used to evaluate computational models of search guidance from three sources: Saliency, target features, and scene context. Each of these models independently outperformed a cross-image control in predicting human fixations. Models that combined sources of guidance ultimately predicted 94% of human agreement, with the scene context component providing the most explanatory power. None of the models, however, could reach the precision and fidelity of an attentional map defined by human fixations. This work puts forth a benchmark for computational models of search in real world scenes. Further improvements in modelling should capture mechanisms underlying the selectivity of observers’ fixations during search.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

In Cognitive Science & Psychology

Citations in the corpus

Extreme Clicking for Efficient Object Annotation (2017, neutral): “Interestingly, 1s visual search is consistent with earlier findings [18, 46].”
Visual search (2010, methodology): “Using measures of eye movements (Ehinger, Hidalgo-Sotelo, Torralba, & Oliva, 2009) or measurements of the deployment of a magnifier (Ehinger & Wolfe, 2015), one can see this guidance at work in tasks as varied as search for humans in natural scenes (Ehinger et al., 2009) or for gas stations in overh”
Eye tracking assisted extraction of attentionally important objects from videos (2015, methodology): “The proposed technique outperforms state-of-the-art video segmentation using eye tracking prior and obtains favorable object extraction over algorithms which do not utilize eye tracking data.”
Action from Still Image Dataset and Inverse Optimal Control to Learn Task Specific Visual Scanpaths (2013, background): “We find good inter-subject agreement for both tasks, consistent with previously reported results for both images and video [14, 19].”
Attentive Systems: A Survey (2017, background): “Ehinger et al. (2009) present a model of search guidance that combines saliency, target features, and scene context, and accounts for 94% of the agreement between human observers searching for targets in over 900 scenes.”
Studying Relationships between Human Gaze, Description, and Computer Vision (2013, background): “The clearest examples of this come from the extensive literature on eye movements during visual search [8, 21, 34, 35]; specifying different targets yields different patterns of eye movements even for the same image.”
Learning to predict where humans look (2009, background): “[5] show that a model of search guidance combining three sources: low level saliency, target features, and scene context, outperforms models based on any of these single sources.”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, result): “Interestingly, 1s visual search is consistent with earlier findings [18, 46].”
Eye guidance in natural vision: Reinterpreting salience (2011, background): “Contextual guidance and low-level features combine to provide good predictive power for human fixation distributions (Ehinger et al., 2009).”
You look where I look! Effect of gaze cues on overt and covert attention in misdirection (2009, background): “Current models, derived mainly from static scene viewing, favour a view in which higher level factors such as behavioural task (Einhauser, Rutishauser, & Koch, 2008 ), expertise with similar scenes (Underwood, Foulsham, & Humphrey, 2009 , this issue), spatial expectation for object location (Ehing”
A Benchmark of Computational Models of Saliency to Predict Human Fixations (2012, methodology): “This research has since been updated by Ehinger et al. [17].”
TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking (2015, background): “he number of subjects, the number of views per image, the subjects distance from the screen, eye trackers, the exact task the subjects were given (free viewing [16], object search [26], person search [9], image rating [37] or memory task [15]), but each helps us understand where people actually look”
Dynamic Eye Movement Datasets and Learnt Saliency Models for Visual Action Recognition (2012, methodology): “These values are consistent with the results reported for static stimuli by [11], with slightly higher cross-stimulus control.”
Visual search: a retrospective. (2011, background): “This is arguably the most prevalent type of model and encompasses a great number of proposed models (Beutter et al., 2003; Eckstein et al., 2006; Ehinger et al., 2009; Findlay, 1997; Pomplun, 2006; Rao, Zelinsky, Hayhoe, & Ballard, 2002; Zelinsky, 2008).”
Scene context influences without scene gist: Eye movements guided by spatial associations in visual search (2011, background): “For instance, in a recent computational model of eye movements during search in real-world scenes, Ehinger, Hidalgo-Sotelo, Torralba, and Oliva (2009) operationalized scene context as the learned association between target placement and the scene’s visual features (e.g., scale, layout, and viewpoi”