TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking

By Pingmei Xu, Krista A. Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, Jianxiong Xiao. arXiv.org, 2015

425 citations recorded by the source corpus.

Traditional eye tracking requires specialized hardware, which means collecting gaze data from many observers is expensive, tedious and slow. Therefore, existing saliency prediction datasets are order-of-magnitudes smaller than typical datasets for other vision recognition tasks. The small size of these datasets limits the potential for training data intensive algorithms, and causes overfitting in benchmark evaluation. To address this deficiency, this paper introduces a webcam-based gaze tracking system that supports large-scale, crowdsourced eye tracking deployed on Amazon Mechanical Turk (AMTurk). By a combination of careful algorithm and gaming protocol design, our system obtains eye tracking data for saliency prediction comparable to data gathered in a traditional lab setting, with relatively lower cost and less effort on the part of the researchers. Using this tool, we build a saliency dataset for a large number of natural images. We will open-source our tool and provide a web server where researchers can upload their images to get eye tracking results from AMTurk.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/3433627f803953280b66ae1576d083fc9a68385a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Human-Computer Interaction

Citations in the corpus

Energy-based Out-of-distribution Detection (2020, methodology): “Out-of-distribution Datasets For the OOD test dataset D testout , we use six common benchmarks: Textures [5], SVHN [29], Places365 [45], LSUN-Crop [42], LSUN-Resize [42], and iSUN [41].”
Gaze360: Physically Unconstrained Gaze Estimation in the Wild (2019, background): “Support vector regression [28], random forests [11] and most recently deep learning [4, 12, 30, 31, 34] have been applied in this way.”
Principled Detection of Out-of-Distribution Examples in Neural Networks (2017, background): “The iSUN [31] consists of a subset of SUN images.”
Process-Tracing Methods in Decision Making: On Growing Up in the 70s (2017, background): “Scaling up this idea, it is also possible to simply use an available webcam on a participant’s computer and access this information to track gaze for large samples online (Xu et al., 2015).”
Learning Confidence for Out-of-Distribution Detection in Neural Networks (2018, methodology): “iSUN: The iSUN dataset (Xu et al., 2015) is a subset of the SUN dataset, containing 8,925 images.”
Seeing with Humans: Gaze-Assisted Neural Image Captioning (2016, background): “Recent advances in crowd-sourcing techniques [6], [7], [8] and appearance-based estimation methods [9], [10], [11] are also further paving the way for low-cost and large-scale gaze data collection.”
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation (2017, background): “visual saliency prediction, such as the crowd-sourced iSUN dataset [57], but their focus is on bottom-up saliency prediction, and input face or eye images are not available.”
Eye Tracking for Everyone (2016, methodology): “3 of [41])—we use the fixed locations to study the effect”
VOS: Learning What You Don't Know by Virtual Outlier Synthesis (2022, methodology): “We evaluate on six OOD datasets: Textures (Cimpoi et al., 2014), SVHN (Netzer et al., 2011), Places365 (Zhou et al., 2018), LSUN-C (Yu et al., 2015), LSUN-Resize (Yu et al., 2015), and iSUN (Xu et al., 2015).”
Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers (2018, methodology): “We perform the ablation studies on DenseNet-BC [8] network with CIFAR-100 [10] as in-distribution while training and iSUN [20] as the OOD validation data while testing.”
Shallow and Deep Convolutional Networks for Saliency Prediction (2016, methodology): “iSUN [43] The iSUN dataset has been built with an online game using webcams to track player eye gaze.”
Generative Probabilistic Novelty Detection with Adversarial Autoencoders (2018, methodology): “For outliers are used samples from TinyImageNet [50], LSUN [51], and iSUN [52].”
CODEs: Chamfer Out-of-Distribution Examples against Overconfidence Issue (2021, methodology): “Specifically, we train WRN-40-2 [49] on CIFAR-10 and CIFAR-100 [24], and then test it on six datasets: Textures [6], SVHN, Places365 [50], LSUN-Crop [47], LSUNResize [47], and iSUN [45] following [32].”
Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy (2019, methodology): “The iSUN is a subset of SUN [28], which is used for gaze tracking, deployed on Amazon Mechanical Turk using a webcam [29].”
End-to-end Convolutional Network for Saliency Prediction (2015, methodology): “Our work has benefited from the recent publication of two datasets: iSun [21] and SALICON [12].”
Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks (2017, background): “The iSUN (Xu et al., 2015) consists of a subset of SUN images.”
Mitigating Neural Network Overconfidence with Logit Normalization (2022, methodology): “Here we use LSUN-C and LSUN-R to denote the cropped and resized version of the LSUN dataset respectively. iSUN (Xu et al., 2015) is a large-scale eye tracking dataset, selected from natural scene images of the SUN database (Xiao et al., 2010).”
ReAct: Out-of-distribution Detection With Rectified Activations (2021, methodology): “For OOD data, we consider six common benchmark datasets: Textures [5], SVHN [37], Places365 [62], LSUN-Crop [60], LSUN-Resize [60], and iSUN [57].”
Out-of-distribution Detection with Deep Nearest Neighbors (2022, methodology): “We evaluate the methods on common OOD datasets: Textures (Cimpoi et al., 2014), SVHN (Netzer et al., 2011), Places365 (Zhou et al., 2017), LSUN-C (Yu et al., 2015), and iSUN (Xu et al., 2015).”