Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks

By Zhe He, Adrian Spurr, Xucong Zhang, Otmar Hilliges. 2019

55 citations recorded by the source corpus.

Gaze redirection is the task of changing the gaze to a desired direction for a given monocular eye patch image. Many applications such as videoconferencing, films, games, and generation of training data for gaze estimation require redirecting the gaze, without distorting the appearance of the area surrounding the eye and while producing photo-realistic images. Existing methods lack the ability to generate perceptually plausible images. In this work, we present a novel method to alleviate this problem by leveraging generative adversarial training to synthesize an eye image conditioned on a target gaze direction. Our method ensures perceptual similarity and consistency of synthesized images to the real images. Furthermore, a gaze estimation loss is used to control the gaze direction accurately. To attain high-quality images, we incorporate perceptual and cycle consistency losses into our architecture. In extensive evaluations we show that the proposed method outperforms state-of-the-art approaches in terms of both image quality and redirection precision. Finally, we show that generated images can bring significant improvement for the gaze estimation task if used to augment real training data.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Generative Models, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Siyu Tang
2. Otmar Hilliges

In Computer Vision and Pattern Recognition

1. Yujiu Yang
2. Weihao Xia
3. Jing-Hao Xue
4. Wensen Feng
5. Derek Bradley
6. Otmar Hilliges
7. Siyu Tang
8. Seonwook Park

In Generative Models

1. Yujiu Yang
2. Weihao Xia
3. Jing-Hao Xue
4. Seonwook Park
5. Xucong Zhang
6. Otmar Hilliges
7. Daniel Cohen-Or

In Human-Computer Interaction

1. Jing-Hao Xue
2. Yujiu Yang
3. Weihao Xia
4. Wensen Feng
5. Otmar Hilliges
6. Derek Bradley
7. Siyu Tang
8. Seonwook Park

Citations in the corpus

• Controllable Continuous Gaze Redirection (2020, methodology): “ly re-render the entire input region by performing 3D transformations, which requires heavy instrumentation to acquire the depth information [9, 19, 39, 45]. Learningbased gaze redirection approaches [10, 13] has risen in recent years. For example, DeepWarp [10] uses a neural network to warp the inp”
• TediGAN: Text-Guided Diverse Image Generation and Manipulation (2020, unknown): “Those conditioned on control vectors are typically limited to a small range of applications such as gaze redi-rection [13] or relighting [44, 59].”
• ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation (2020, methodology): “Some of these approaches do result in lower resolution images, and as such are not informative in the development of generative models [30] or gaze redirection methods [12,38].”