Predicting human gaze beyond pixels.

By Juan Xu, Ming Jiang, Shuo Wang, Mohan Kankanhalli, Qi Zhao. 2014

360 citations recorded by the source corpus.

A large body of previous models to predict where people look in natural scenes focused on pixel-level image attributes. To bridge the semantic gap between the predictive power of computational saliency models and human behavior, we propose a new saliency architecture that incorporates information at three layers: pixel-level image attributes, object-level attributes, and semantic-level attributes. Object- and semantic-level information is frequently ignored, or only a few sample object categories are discussed where scaling to a large number of object categories is not feasible nor neurally plausible. To address this problem, this work constructs a principled vocabulary of basic attributes to describe object- and semantic-level information thus not restricting to a limited number of object categories. We build a new dataset of 700 images with eye-tracking data of 15 viewers and annotation data of 5,551 segmented objects with fine contours and 12 semantic attributes (publicly available with the paper). Experimental results demonstrate the importance of the object- and semantic-level information in the prediction of visual attention.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Sensory Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Sensory Systems

Citations in the corpus

What Do Different Evaluation Metrics Tell Us About Saliency Models? (2016, background): “Additionally, consistency between fixations of different subjects is high just after stimulus onset but progressively decreases over time [75], [87].”
Seeing with Humans: Gaze-Assisted Neural Image Captioning (2016, background): “The correlation between ﬁxations and objects is also becoming important in the context of saliency prediction [24], [25].”
SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks (2015, methodology): “The descriptions of these datasets are listed below: - OSIE [40]: This dataset contains 700 images with annotated objects and attributes content.”
Top-down visual saliency via joint CRF and dictionary learning (2012, background): “However, as target objects are likely to appear anywhere in a scene, this contextual prior is less effective (e.g., images from the Graz-02 and PASCAL VOC datasets).”
SALICON: Saliency in Context (2015, methodology): “Human labeling such as object bounding boxes [16], contours [27, 35], and social attributes [35, 16] are available in certain datasets as ground truth data for learning and analysis of problems of interest.”
Human Gaze Assisted Artificial Intelligence: A Review (2020, neutral): “…approaches became more popular as large-scale eye-tracking datasets became available for images [Papadopoulos et al. , 2014; Li et al. , 2014; Xu et al. , 2014; Bylinskii et al. , 2015b; Bylinskii et al. , 2015a; Krafka et al. , 2016], videos [Mathe and Sminchisescu, 2014; Wang et al. , 2018…”
DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations (2015, methodology): “We evaluate the proposed network on multiple challenging saliency datasets MIT300 [22], CAT2000 [23], PASCAL-S [24], OSIE [25], FIGRIM [26], SALICON Test Set [27], iSUN Test Set and show that it achieves state-of-theart results on these datasets.”
TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking (2015, background): “There are other datasets focused on specific domains: PASCAL-S [19] offers both fixation and salient object segmentation; OSIE [41] features in multiple dominant objects in each image; and the MIT Low Resolution dataset [14] contains low resolution images.”