DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation

By Yaroslav Ganin, Daniil Kononenko, Diana Sungatullina, Victor S. Lempitsky. 2016

128 citations recorded by the source corpus.

In this work, we consider the task of generating highly-realistic images of a given face with a redirected gaze. We treat this problem as a specific instance of conditional image generation and suggest a new deep architecture that can handle this task very well as revealed by numerical comparison with prior art and a user study. Our deep architecture performs coarse-to-fine warping with an additional intensity correction of individual pixels. All these operations are performed in a feed-forward manner, and the parameters associated with different operations are learned jointly in the end-to-end fashion. After learning, the resulting neural network can synthesize images with manipulated gaze, while the redirection angle can be selected arbitrarily from a certain range and provided as an input to the network.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

GAN Inversion: A Survey (2021, background): “, gaze redirection [7], [231], [232], [233], relighting [234], [235], [236] and continuous view control [237].”
AdaptIS: Adaptive Instance Selection Network (2019, neutral): “in [9].”
GazeDirector: Fully Articulated Eye Gaze Redirection in Video (2017, methodology): “Comparison to DeepWarp [12] Previous work produces unsightly smudging artefacts when starting from non-central gaze, and redirecting gaze over large angles.”
Animating Arbitrary Objects via Deep Motion Transfer (2018, background): “However, standard convolutional encoder-decoders are not designed to handle large pixel-to-pixel misalignment between the input and output images [31, 3, 14].”
The Creation and Detection of Deepfakes (2020, methodology): “, [79, 135, 145, 162, 172, 174]) are more likely to be used by researchers and criminals compared to those which are unavailable [2, 55, 65, 83, 121, 127, 154, 171, 185] or require highly specic or private datasets [57, 113, 181].”
Video Frame Synthesis Using Deep Voxel Flow (2017, methodology): “Optical flow layers have also been used to render novel views of objects [38, 14] and change eye gaze direction while videoconferencing [8].”
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation (2016, methodology): “…can be expressed as I ′ t +1 ( x, y ) = I{ I t +1 ( x + ∆ t +1 x, y + ∆ t +1 y ) , or more concisely We adopt a multi-scale design to represent the ﬂow, which has been shown to be effective in classical methods [10, 2] and also in more recently proposed spatial trans-former techniques [11, 1, 9].”
Eye In-painting with Exemplar Generative Adversarial Networks (2017, methodology): “Besides classic computer vision techniques, recent research has focused on using deep convolutional networks to perform a variety of facial transformations [11].”
Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks (2019, methodology): “In previous work of monocular gaze manipulation [8, 29], the mean squared error (MSE) was used as the metric to measure the similarity between the generated eye images and ground-truth eye images.”
GAN-Supervised Dense Visual Alignment (2021, background): “, classification) and applications such as robust filter learning [16, 37], view synthesis [26, 66, 94] and 3D representation learning [39, 87, 92].”
Neural Point-Based Graphics (2019, background): “Estimating warping fields from a single input image and a low-dimensional parameter specifying a certain motion from a low-parametric family is also possible [24,25].”
Controllable Continuous Gaze Redirection (2020, methodology): “For example, DeepWarp [10] uses a neural network to warp the input image to the desired direction by predicting the dense flow field.”
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models (2019, background): “Both classical warping algorithms [5, 28] and warping fields synthesized using machine learning (including deep learning) [11, 29, 40] can be used for such purposes.”