Rendering of Eyes for Eye-Shape Registration and Gaze Estimation

By Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, Andreas Bulling. IEEE International Conference on Computer Vision, 2015

347 citations recorded by the source corpus.

Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniquesto build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model's controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/5e9f10adb1596259c7e83fce337575b11381e6c7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Human-Computer Interaction

Citations in the corpus

Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “Instead of collecting and labelling real data, which is slow, expensive, and subject to bias, it can be preferable to synthesize training data using computer graphics [68].”
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation (2017, methodology): “As discussed in [15] and [14], one of the most important challenges for unconstrained gaze estimation is differences in gaze ranges between the training and testing domains.”
OpenFace: An open source facial behavior analysis toolkit (2016, methodology): “We used the SynthesEyes training dataset [62] to train the PDM and”
OpenFace 2.0: Facial Behavior Analysis Toolkit (2018, methodology): “In order to estimate eye gaze, we use a Constrained Local Neural Field (CLNF) landmark detector [9], [70] to detect eyelids, iris, and the pupil.”
Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “…neural networks have been shown to have better tracking accuracy and are more robust to visual appearance variations [Zhang et al. , 2015; Wood et al. , 2015; Krafka et al. , 2016; Shrivastava et al. , 2017; Zhang et al. , 2017; Park et al. , 2018], compared to more traditional…”
Controllable Continuous Gaze Redirection (2020, methodology): “, SynthesEyes [37], UnityEyes [34]), or use limited high-quality dataset [13] (e.”
Synthetic Data for Deep Learning (2019, methodology): “[640,641] presented a large dataset of realistic renderings of human eyes and showed improvements on real test sets over previous work done with the MPIIgaze dataset of real labeled images [694].”
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos (2016, background): “Visual features include 16 Facial Action Units, 68 Facial Land-marks, Head Pose and Orientation, 6 Basic Emotions 6 and Eye Gaze (Wood et.al., 2015, Baltru-saitis et.al., 2012, Baltrusaitis et.al., 2014).”
RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments (2018, background): “X Frontal SynthesEyes [56] RGB Synthesized eye patches Gaze vector 11,382 Varying X X All UnityEyes [55] RGB Synthesized eye patches Gaze vector 1,000,000 Varying X X All UT Multi-view [53] RGB Eye area + eye patches Gaze vector 1,152,000 60cm X X All Gaze Capture [28] RGB Face + eye patches 2D pos ”
It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation (2016, methodology): “Consequently, recent years have seen an increasing number of gaze estimation datasets collected in everyday settings [9, 19, 24], including some at large scale [13, 36], or consisting of synthetic data [25, 32, 33].”