Eye guidance in natural vision: Reinterpreting salience

By B. Tatler, Mary M. Hayhoe, M. Land, Dana H. Ballard. 2011

776 citations recorded by the source corpus.

Models of gaze allocation in complex scenes are derived mainly from studies of static picture viewing. The dominant framework to emerge has been image salience, where properties of the stimulus play a crucial role in guiding the eyes. However, salience-based schemes are poor at accounting for many aspects of picture viewing and can fail dramatically in the context of natural task performance. These failures have led to the development of new models of gaze allocation in scene viewing that address a number of these issues. However, models based on the picture-viewing paradigm are unlikely to generalize to a broader range of experimental contexts, because the stimulus context is limited, and the dynamic, task-driven nature of vision is not represented. We argue that there is a need to move away from this class of model and find the principles that govern gaze allocation in a broader range of settings. We outline the major limitations of salience-based selection schemes and highlight what we have learned from studies of gaze allocation in natural vision. Clear principles of selection are found across many instances of natural vision and these are not the principles that might be expected from picture-viewing studies. We discuss the emerging theoretical framework for gaze allocation on the basis of reward maximization and uncertainty reduction.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Molecular Biology

Citations in the corpus

Predicting the Driver's Focus of Attention: The DR(eye)VE Project (2017, background): “According to [54], the act of driving combines complex attention mechanisms guided by the driver’s past experience, short reactive times and strong contextual constraints.”
Efficiently Guiding Imitation Learning Algorithms with Human Gaze (2020, background): “Human attention in the form of eye gaze has been known to encode top-down attention versus bottom-up salience when performing goal directed tasks (Land 2009; Hayhoe and Ballard 2005; Rothkopf, Ballard, and Hayhoe 2007; Tatler et al. 2011).”
Review of Eye Tracking Metrics Involved in Emotional and Cognitive Processes (2021, background): “Although salience-based schemes are still widely used in computational models, they prove to be poor in predicting eye movement behaviour in natural tasks [80].”
Perceptions as Hypotheses: Saccades as Experiments (2012, background): “Saccadic eye movements are a useful vehicle to illustrate active inference about salient features of the world because they speak directly to visual search strategies and a wealth of psychophysical, neurobiological, and theoretical study (e.g., Grossberg et al., 1997; Ferreira et al., 2008; Srihasam”
Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study (2013, background): “[89]), retinal sampling, inhibition of return, the role of context, etc [90].”
What do eyes reveal about the mind?: Algorithmic inference of search targets from fixations (2015, neutral): “Please see [1] [33] [78] [74] [57] [48] [47] [39] [5] for reviews.”
Visual search: a retrospective. (2011, background): “…of eye movement endpoints when human observers are engaged in a speciﬁc task such as search (Einha ¨ user, Rutishauser, & Koch, 2008; Foulsham & Underwood, 2008; Pomplun, 2006; Torralba et al., 2006; see, in this special anniversary issue of JOV, Tatler et al., 2011 for a review on saliency).”
Attentive Systems: A Survey (2017, background): “As studied in Tatler et al. (2011), there is a consistent set of principles underlying search guidance involving behavioral relevance, reward, or uncertainty about the state of the environment, as well as the learned models of the environment, or priors.”
Attention and choice: a review on eye movements in decision making. (2013, background): “However, the critical question is, how do decision makers know what stimuli are task relevant? One possibility is that decision makers learn through practice to distinguish relevant from irrelevant stimuli (Haider & Frensch, 1999); a process which could be driven by feedback about the reward value o”