Eye Tracking for Everyone
By Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, Antonio Torralba. Computer Vision and Pattern Recognition, 2016
997 citations recorded by the source corpus.
From scientific research to commercial applications, eye tracking is an important tool across many domains. Despite its range of applications, eye tracking has yet to become a pervasive technology. We believe that we can put the power of eye tracking in everyone's palm by building eye tracking software that works on commodity hardware such as mobile phones and tablets, without the need for additional sensors or devices. We tackle this problem by introducing GazeCapture, the first large-scale dataset for eye tracking, containing data from over 1450 people consisting of almost 2:5M frames. Using GazeCapture, we train iTracker, a convolutional neural network for eye tracking, which achieves a significant reduction in error over previous approaches while running in real time (10-15fps) on a modern mobile device. Our model achieves a prediction error of 1.71cm and 2.53cm without calibration on mobile phones and tablets respectively. With calibration, this is reduced to 1.34cm and 2.12cm. Further, we demonstrate that the features learned by iTracker generalize well to other datasets, achieving state-of-the-art results. The code, data, and models are available at http://gazecapture.csail.mit.edu.
Subfields: Radiology, Nuclear Medicine and Imaging, Urban Studies, Human-Computer Interaction
View Paper: https://www.semanticscholar.org/paper/0695751eb18cd138d7d9441378739882a8afc919
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Radiology, Nuclear Medicine and Imaging
- Xucong Zhang
- Mario Fritz
- Andreas Bulling
- Yusuke Sugano
- Yiannis Demiris
- Chi Zhang
- Tobias Fischer
- Hyung Jin Chang
In Urban Studies
In Human-Computer Interaction
- Xucong Zhang
- Andreas Bulling
- Ruohan Zhang
- Mario Fritz
- Yusuke Sugano
- Yiannis Demiris
- Antonio Torralba
- Hyung Jin Chang
Citations in the corpus
- Human Gaze Assisted Artificial Intelligence: A Review (2020, methodology): “…became available for images [Papadopoulos et al. , 2014; Li et al. , 2014; Xu et al. , 2014; Bylinskii et al. , 2015b; Bylinskii et al. , 2015a; Krafka et al. , 2016], videos [Mathe and Sminchisescu, 2014; Wang et al. , 2018 ] , and 360-degree videos [ Zhang et al. , 2018b; Xu et al. , 2018].”
- RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments (2018, methodology): “GazeCapture [28] is a crowd-sourced dataset of nearly 1500 subjects looking at gaze targets on a tablet screen.”
- SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (2017, background): “These data are traditionally measured with eye-trackers, and more recently with mouse clicks [12] or webcams [15].”
- Controllable Continuous Gaze Redirection (2020, background): “CMU Multi-Pie [11] ✓ ✗ ✓ Facial landmarks 755,370 ✓ Small Gaze Capture [17] ✓ ✗ ✗ 2D position on screen >2.”
- DEEP eye contact detector: Robust eye contact bid detection using convolutional neural network (2017, background): “Krafka et al. developed a DNN-based eye-gaze-estimation algorithm that inputs full-facial images as well as eye images [9].”
- Gaze360: Physically Unconstrained Gaze Estimation in the Wild (2019, methodology): “A preprocessing step of eye or face detection is often required [12, 30].”
- ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation (2020, background): “In [19], the authors show gaze estimation performance with different numbers of participants.”
- Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze (2020, background): “As 3D gaze direction of a person is an important cue for understanding human attention, 3D gaze estimation task has received significant attention from the computer vision community (Brau et al. 2018; Huang et al. 2017; Krafka et al. 2016; Zhang et al. 2015, 2017a; Zhu and Deng 2017).”
- Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks (2018, methodology): “n GazeCapture dataset. The table reports distance errors in cm for our models and benchmarks on the GazeCapture dataset. We choose AlexNet in order to be directly comparable to the iTracker system of [21], which is one of the state-of-the-art models for gaze tracking. The model has four inputs: two ”
- EmotionMeter: A Multimodal Framework for Recognizing Human Emotions (2019, background): “Eye tracking using wearable techniques has received considerable attention in recent years due to its natural observations and informative features of users’ nonverbal behaviors [38], [73], [74].”
- AGIL: Learning Attention from Human for Visuomotor Tasks (2018, background): “In recent years, this approach has made tremendous progress due to large benchmark datasets and deep neural networks [19, 5, 25, 22, 42, 21].”
- MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation (2017, methodology): “Prior work performed person-independent gaze estimation using 2D regression in the screen coordinate system [17], [18].”
- It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation (2016, methodology): “While more recent methods use 3D head pose [18, 26] or size and location of the face bounding box [13] to allow for free head movement, they still formulate the task as a direct mapping to 2D on-screen gaze locations.”
- Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks (2019, background): “Furthermore, learning-based gaze estimation has recently made significant progress based on in-the-wild datasets [18, 33].”
- Efficient eye typing with 9-direction gaze estimation (2017, methodology): “A big advantage of using CNN to estimate gaze is that CNN methods do not require complex calibration to determine a series of parameters, and in some CNN based methods, calibration is even not needed [5].”
- Person Recognition System Based on a Combination of Body Images from Visible Light and Thermal Cameras (2017, methodology): “Later, various image processing systems such as face recognition [8], person re-identification [14,15,25], gaze estimation [30], face detection [31], eye tracking [32], and lane detection [33] were solved by using a deep learning framework with high performance.”