Appearance-based gaze estimation in the wild

By Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling. 2015

924 citations recorded by the source corpus.

Appearance-based gaze estimation is believed to work well in real-world settings, but existing datasets have been collected under controlled laboratory conditions and methods have been not evaluated across multiple datasets. In this work we study appearance-based gaze estimation in the wild. We present the MPIIGaze dataset that contains 213,659 images we collected from 15 participants during natural everyday laptop use over more than three months. Our dataset is significantly more variable than existing ones with respect to appearance and illumination. We also present a method for in-the-wild appearance-based gaze estimation using multimodal convolutional neural networks that significantly outperforms state-of-the art methods in the most challenging cross-dataset evaluation. We present an extensive evaluation of several state-of-the-art image-based gaze estimation algorithms on three current datasets, including our own. This evaluation provides clear insights and allows us to identify key research challenges of gaze estimation in the wild.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Human-Computer Interaction

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, methodology): “We evaluate our method for appearance-based gaze estimation in the wild on the MPIIGaze dataset [43, 47], and hand pose estimation on the NYU hand pose dataset of depth images [38].”
A Domain Based Approach to Social Relation Recognition (2017, methodology): “This double-stream model is similar to multi-region CNNs [37], based on LeNet [43].”
Human Gaze Assisted Artificial Intelligence: A Review (2020, methodology): “…algorithms using convolutional neural networks have been shown to have better tracking accuracy and are more robust to visual appearance variations [Zhang et al. , 2015; Wood et al. , 2015; Krafka et al. , 2016; Shrivastava et al. , 2017; Zhang et al. , 2017; Park et al. , 2018], compared to more…”
AI Benchmark: Running Deep Neural Networks on Android Smartphones (2018, background): “nhancement [4,5,6] and super-resolution [7,8,9], optical character recognition [10], object tracking [11,12], visual scene understanding [13,14], face detection and recognition [15,16], gaze tracking [17], etc. Another group of tasks encompasses various natural language processing problems such as n”
Synthetic Data for Deep Learning (2019, methodology): “report performance improvements of the model itself applied to gaze estimation over [545] for sufficiently large training sets, and also show that the synthetic data generated by the model improves the results of standard gaze estimators (LeNet, as used in [694]).”
Production-level facial performance capture using deep convolutional neural networks (2016, background): “Specialized techniques have also been proposed for various subdomains, including eyelids [Bermano et al. 2015], gaze direction [Zhang et al. 2015], lips [Garrido et al. 2016], handling of occlusions [Saito et al. 2016], and handling of extreme skin deformations [Wu et al. 2016].”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “Our training setup matches the accuracies reported in the original works [67, 68].”
OpenFace 2.0: Facial Behavior Analysis Toolkit (2018, methodology): “We evaluated the ability of OpenFace 2.0 to estimate eye gaze vectors by evaluating it on the challenging MPIIGaze dataset [79] intended to evaluate appearance based gaze estimation.”
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation (2017, methodology): “18, while head pose information does result in a performance improvement for the shallower MnistNet architecture used in [15], it does not significantly improve the performance of GazeNet.”
Eye Tracking for Everyone (2016, methodology): “While some of the modern datasets follow a similar approach [34, 25, 43, 13], their scale—especially in the number of participants—is rather limited.”
Controllable Continuous Gaze Redirection (2020, background): “Those [17, 43] proposed for gaze estimation are collected under unconstrained environmental conditions and vary in background and illumination, which means these datasets are not suitable for high-quality gaze redirection.”
Gaze360: Physically Unconstrained Gaze Estimation in the Wild (2019, background): “Support vector regression [28], random forests [11] and most recently deep learning [4, 12, 30, 31, 34] have been applied in this way.”
It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation (2016, methodology): “For the MPIIGaze dataset, we performed a leave-one-person-out cross-validation on all 15 participants.”
Attentive Systems: A Survey (2017, background): “Therefore, there are a few approaches (Zhang et al. 2015; Sugano et al. 2010; Choi etal.”
DEEP eye contact detector: Robust eye contact bid detection using convolutional neural network (2017, positive): “proposed a DNN algorithm that inputs eye images and a 3D head pose obtained from facial-landmark points [25].”