Cross-Domain and Disentangled Face Manipulation With 3D Guidance

By Can Wang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao. 2021

28 citations recorded by the source corpus.

Face image manipulation via three-dimensional guidance has been widely applied in various interactive scenarios due to its semantically-meaningful understanding and user-friendly controllability. However, existing 3D-morphable-model-based manipulation methods are not directly applicable to out-of-domain faces, such as non-photorealistic paintings, cartoon portraits, or even animals, mainly due to the formidable difficulties in building the model for each specific face domain. To overcome this challenge, we propose, as far as we know, the first method to manipulate faces in arbitrary domains using human 3DMM. This is achieved through two major steps: 1) disentangled mapping from 3DMM parameters to the latent space embedding of a pre-trained StyleGAN2 [1] that guarantees disentangled and precise controls for each semantic attribute; and 2) cross-domain adaptation that bridges domain discrepancies and makes human 3DMM applicable to out-of-domain faces by enforcing a consistent latent space embedding. Experiments and comparisons demonstrate the superiority of our high-quality semantic manipulation method on a variety of face domains with all major 3D facial attributes controllable – pose, expression, shape, albedo, and illumination. Moreover, we develop an intuitive editing interface to support user-friendly control and instant feedback. Our project page is https://cassiepython.github.io/cddfm3d/index.html.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

StyleGAN-NADA (2021, methodology): “First, due to our intertwined generator architecture and the nature of the directional loss – and second, because prior methods successfully employed the natural alignment of a finetuned generator for downstream applications [31, 37, 44].”
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion (2021, methodology): “In this paper, we follow existing methods [41], [10], [4], [5], [7], [11] and choose StyleGAN as the inversion target, but the proposed feed-forward GAN inversion network and the accompanying design principles can be easily adapted to other GAN models.”