Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning

By Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong. 2020

394 citations recorded by the source corpus.

We propose an approach for face image generation of virtual people with disentangled, precisely-controllable latent representations for identity of non-existing people, expression, pose, and illumination. We embed 3D priors into adversarial learning and train the network to imitate the image formation of an analytic 3D face deformation and rendering process. To deal with the generation freedom induced by the domain gap between real and rendered faces, we further introduce contrastive learning to promote disentanglement by comparing pairs of generated images. Experiments show that through our imitative-contrastive learning, the factor variations are very well disentangled and the properties of a generated face can be precisely controlled. We also analyze the learned latent space and present several meaningful properties supporting factor disentanglement. Our method can also be used to embed real images into the disentangled latent space. We hope our method could provide new understandings of the relationship between physical properties and deep image synthesis.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “It has been explored under several adversarial training scenarios [25, 65, 9, 41].”
Knowledge Graph Contrastive Learning for Recommendation (2022, background): “Recently,contrastive learning has attracted much attention in offering self-supervised signals for various domains, nature language processing [8] and image data analysis [6].”
Motion Representations for Articulated Animation (2021, background): “Such knowledge typically includes landmarks [4, 40, 24, 12], semantic segmentations [22], and parametric 3D models [11, 32, 9, 20, 18].”
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection (2021, background): “However, with remarkable progress made in facial synthesis technology [20, 45, 12], such meth-ods have been unable to reliably detect face forgery.”
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search (2021, methodology): “Recently, 3D prior ( e.g. 3DMM [Blanz and Vetter, 1999]) is also introduced to encode the synthetic image [Deng et al. , 2020; Tewari et al. , 2020].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Some strategies, such as adding disentangled loss [9, 26, 49], embedding the above-mentioned parametric face model into generative adversarial network (GAN) [5, 14], etc.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Some works proposed solutions on this task [24, 25, 36, 42, 45, 63, 112, 143].”
Unsupervised Discovery of Disentangled Manifolds in GANs (2020, background): “[13] embed 3D prior into adversarial training and generated faces can be precisely controlled.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “Another line of works [15,74] focuses on controllable face genera-D tion, providing explicit control over the generated face from a pretrained StyleGAN [33, 34].”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, methodology): “The regional-specific property of our method also helps in dealing with the entanglement issue in most existing decoupling-based methods for the controllable generation ( [Deng et al. 2020; Shen et al. 2019, 2020]).”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, methodology): “Recently, StyleRig (Tewari et al., 2020), DiscoFaceGAN (Deng et al., 2020), and RotateRender (Zhou et al., 2020) achieves 3D-aware manipulation on human face by using an external 3D human face model, i.e. , 3DMM (Blanz & Vet-ter, 1999), as a strong 3D shape prior.”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “Most previous approaches gain controllability of GANs via prior 3D models [Deng et al. 2020; Ghosh et al. 2020; Tewari et al. 2020] or supervised learning that relies on manually annotated data [Abdal et al. 2021; Isola et al. 2017; Ling et al. 2021; Park et al. 2019; Shen et al. 2020].”