AgileGAN

By Guoxian Song, Linjie Luo, Jing Liu, Wan-Chun Ma, Chun-Pong Lai, Chuanxia Zheng, Tat-Jen Cham. 2021

60 citations recorded by the source corpus.

Portraiture as an art form has evolved from realistic depiction into a plethora of creative styles. While substantial progress has been made in automated stylization, generating high quality stylistic portraits is still a challenge, and even the recent popular Toonify suffers from several artifacts when used on real input images. Such StyleGAN-based methods have focused on finding the best latent inversion mapping for reconstructing input images; however, our key insight is that this does not lead to good generalization to different portrait styles. Hence we propose AgileGAN, a framework that can generate high quality stylistic portraits via inversion-consistent transfer learning. We introduce a novel hierarchical variational autoencoder to ensure the inverse mapped distribution conforms to the original latent Gaussian distribution, while augmenting the original space to a multi-resolution latent space so as to better encode different levels of detail. To better capture attribute-dependent stylization of facial features, we also present an attribute-aware generator and adopt an early stopping strategy to avoid overfitting small training datasets. Our approach provides greater agility in creating high quality and high resolution (1024×1024) portrait stylization models, requiring only a limited number of style exemplars (~100) and short training time (~1 hour). We collected several style datasets for evaluation including 3D cartoons, comics, oil paintings and celebrities. We show that we can achieve superior portrait stylization quality to previous state-of-the-art methods, with comparisons done qualitatively, quantitatively and through a perceptual user study. We also demonstrate two applications of our method, image editing and motion retargeting.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Peihao Zhu
2. Peter Wonka
3. Rameen Abdal
4. Eli Shechtman
5. Dani Lischinski
6. Zongze Wu
7. Jing Liao
8. Yotam Nitzan

In Computer Vision and Pattern Recognition

1. Daniel Cohen-Or
2. Eli Shechtman
3. Dani Lischinski
4. Peter Wonka
5. Ming-Hsuan Yang
6. Yotam Nitzan
7. Peihao Zhu
8. Luc Van Gool

In Generative Models

1. Eli Shechtman
2. Yotam Nitzan
3. Dani Lischinski
4. Peihao Zhu
5. Daniel Cohen-Or
6. Peter Wonka
7. Zongze Wu
8. Rameen Abdal

Citations in the corpus

• Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, methodology): “Regarding the latent-consistent finetuning, the major difference between our method and AgileGAN [Song et al. 2021] is that we innovatively align the style space by fixing the affine transformation layers and finetuning the remaining layers of StyleGAN2 [Karras et al.”
• Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, methodology): “Most relevant to our work, the domain adaptation methods (Patashnik et al., 2021; Gal et al., 2021; Jang et al., 2021; Song et al., 2021) that build upon StyleGAN (Kar-ras et al., 2019; 2020b;a) demonstrate impressive visual quality and semantic interpretability in the target domain.”
• StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, methodology): “Recently, several works (Pinkney & Adler, 2020; bryandlee, 2020; Kwong et al., 2021; Song et al., 2021; Gal et al., 2021) have taken a different approach towards image-to-image translation.”