SofGAN: A Portrait Image Generator with Dynamic Styling

By Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, Jingyi Yu. 2020

93 citations recorded by the source corpus.

Recently, Generative Adversarial Networks (GANs) have been widely used for portrait image generation. However, in the latent space learned by GANs, different attributes, such as pose, shape, and texture style, are generally entangled, making the explicit control of specific attributes difficult. To address this issue, we propose a SofGAN image generator to decouple the latent space of portraits into two subspaces: a geometry space and a texture space. The latent codes sampled from the two subspaces are fed to two network branches separately, one to generate the 3D geometry of portraits with canonical pose, and the other to generate textures. The aligned 3D geometries also come with semantic part segmentation, encoded as a semantic occupancy field (SOF). The SOF allows the rendering of consistent 2D semantic segmentation maps at arbitrary views, which are then fused with the generated texturemaps and stylized to a portrait photo using our semantic instance-wise module. Through extensive experiments, we show that our system can generate high-quality portrait images with independently controllable geometry and texture attributes. The method also generalizes well in various applications, such as appearance-consistent facial animation and dynamic styling.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Ben Mildenhall
2. Christian Theobalt
3. Matthias Nießner
4. Ayush Kumar Tewari
5. Michael Zollhöfer
6. Gordon Wetzstein
7. Justus Thies
8. Vincent Sitzmann

In Computer Graphics and Computer-Aided Design

1. Matthias Nießner
2. Christian Theobalt
3. Ben Mildenhall
4. Andreas Geiger
5. Jonathan T. Barron
6. Gordon Wetzstein
7. Vincent Sitzmann
8. Justus Thies

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Andreas Geiger
4. Jitendra Malik
5. Dacheng Tao
6. Chen Change Loy
7. Matthias Nießner
8. Andrea Vedaldi

In Generative Models

1. Yujiu Yang
2. Weihao Xia
3. Jing-Hao Xue
4. Ben Mildenhall
5. Yulun Zhang
6. Xingang Pan
7. Matthias Nießner
8. Ayush Kumar Tewari

Citations in the corpus

• GAN Inversion: A Survey (2021, methodology): “It has been used for volumetric performance capture [168], [169], [170], novel-view synthesis [171], [171], face shape”
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, background): “With MLPs, however, pixel-wise learnable upsamplers such as pixelshuffle (Shi et al., 2016) or LIIF (Chen et al., 2021b) produce “chessboard” or “texture sticking” artifacts due to its tendency of relying on the image coordinates implicitly.”
• Generative Adversarial Networks for Face Generation: A Survey (2022, background): “SofGAN [20] decouples the latent space of portraits into a geometry space and a texture space.”
• Advances in Neural Rendering (2021, background): “Here, some approaches learn the texture in image space [CLX ∗ 21, XPY ∗ 21], while others learn both geometry and texture in 3D [SWZ ∗ 21, SLNG20].”
• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “Several methods modify the architecture of the GAN to enable 3D control [Chan et al. 2022, 2021; Chen et al. 2022; Gu et al. 2022; Pan et al. 2021; Schwarz et al. 2020; Tewari et al. 2022; Xu et al. 2022].”