Of Bits and Wows: A Bayesian Theory of Surprise with Applications to Attention

By Pierre Baldi, Laurent Itti. Neural Networks, 2009

310 citations recorded by the source corpus.

The amount of information contained in a piece of data can be measured by the effect this data has on its observer. Fundamentally, this effect is to transform the observer's prior beliefs into posterior beliefs, according to Bayes theorem. Thus the amount of information can be measured in a natural way by the distance (relative entropy) between the prior and posterior distributions of the observer over the available space of hypotheses. This facet of information, termed “surprise”, is important in dynamic situations where beliefs change, in particular during learning and adaptation. Surprise can often be computed analytically, for instance in the case of distributions from the exponential family, or it can be numerically approximated. During sequential Bayesian learning, surprise decreases like the inverse of the number of training examples. Theoretical properties of surprise are discussed, in particular how it differs and complements Shannon's definition of information. A computer vision neural network architecture is then presented capable of computing surprise over images and video stimuli. Hypothesizing that surprising data ought to attract natural or artificial attention systems, the output of this architecture is used in a psychophysical experiment to analyze human eye movements in the presence of natural video stimuli. Surprise is found to yield robust performance at predicting human gaze (ROC-like ordinal dominance score ∼ 0.7 compared to ∼ 0.8 for human inter-observer repeatability, ∼ 0.6 for simpler intensity contrast-based predictor, and 0.5 for chance). The resulting theory of surprise is applicable across different spatio-temporal scales, modalities, and levels of abstraction.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/59eb1ea1502c33ef59c4abfc4e8916b490fb538b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Arousal-biased competition in perception and memory (2011, unknown): “Analogous to salience maps for perceptual contrast, a computational model can quantify how surprising each location on a feature map is by comparing beliefs about what is likely to be in that location before and after seeing the information (Baldi & Itti, 2010).”
Learning and exploration in action-perception loops (2013, background): “To model bottom–up visual saliency and predict gaze attention, Itti and Baldi recently developed an information-theoretic measure closely related to PEIG (Itti and Baldi, 2006, 2009; Baldi and Itti, 2010).”
Studying Relationships between Human Gaze, Description, and Computer Vision (2013, background): “To the extent that gaze is drawn to oddities and inconsistencies in a scene [28], fixations might also serve to predict unusual events [1].”
Information Gaps: A Theory of Preferences Regarding the Presence and Absence of Information (2018, background): “Itti and Baldi (2009; Baldi & Itti, 2010) show that surprise, specified this way, predicts the level of attention paid to information (see also Meyer et al., 1991).”
Visual adaptation and novelty responses in the superior colliculus (2011, methodology): “This differs from the Itti and Baldi implementation (Itti & Baldi, 2005; Baldi & Itti, 2010), where the KL divergence is computed at each learning stage, and the system output is the product of the outputs at each stage.”
Attention, Uncertainty, and Free-Energy (2010, background): “Using a more descriptive approach, Itti and Baldi (2006; 2009) proposed that many factors, which influence visual salience, can be integrated with prior expectations by calculating Bayesian surprise (Baldi and Itti, 2010).”
When the world becomes 'too real': a Bayesian explanation of autistic perception. (2012, background): “surprise” [65]?  How can this account be related to Bayesian models of other neuropsychiatric conditions? For example, in schizophrenia, hallucinations are assumed to result from hyper-priors [66, 67], which would place autistic and schizophrenic symptoms at the extremes of a continuum.”