Semantically Multi-Modal Image Synthesis

By Zhen Zhu, Zhi-liang Xu, Ansheng You, Xiang Bai. 2020

103 citations recorded by the source corpus.

In this paper, we focus on semantically multi-modal image synthesis (SMIS) task, namely, generating multi-modal images at the semantic level. Previous work seeks to use multiple class-specific generators, constraining its usage in datasets with a small number of classes. We instead propose a novel Group Decreasing Network (GroupDNet) that leverages group convolutions in the generator and progressively decreases the group numbers of the convolutions in the decoder. Consequently, GroupDNet is armed with much more controllability on translating semantic labels to natural images and has plausible high-quality yields for datasets with many classes. Experiments on several challenging datasets demonstrate the superiority of GroupDNet on performing the SMIS task. We also show that GroupDNet is capable of performing a wide range of interesting synthesis applications. Codes and models are available at: https://github.com/Seanseattle/SMIS.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Bernt Schiele
4. Chen Change Loy
5. Xiaogang Wang
6. Dacheng Tao
7. Jitendra Malik
8. Trevor Darrell

In Multimodal Learning

1. Yujiu Yang
2. Weihao Xia
3. Jing-Hao Xue
4. Ming-Hsuan Yang
5. Baoyuan Wu
6. Bolei Zhou
7. Juergen Gall
8. Yulun Zhang

Citations in the corpus

• GAN Inversion: A Survey (2021, methodology): “The StyleGAN-based architectures have been applied to numerous applications [38], [39], [40].”
• TediGAN: Text-Guided Diverse Image Generation and Manipulation (2020, background): “, label [64, 63], sketch [65, 16], text [53, 36], or control vector like gaze [13]”
• You Only Need Adversarial Supervision for Semantic Image Synthesis (2020, methodology): “Different from structured noise injection of Alharbi & Wonka (2020) and class-specific latent codes of Zhu et al. (2020), we inject 3D noise along with label maps and adjust it to image resolution, enabling also re-sampling of selected semantic segments (see Fig.”
• Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, background): “, label [20], [21], sketch [22], [23], text [24], [25], or control vector like gaze [26] or light direction [27].”
• Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “The input semantic masks are the same as those used in Figure 7. ity [6, 22, 23, 32, 31], generate multi-modal outputs [23, 18, 44, 10], and simplify input annotations using bounding boxes [41, 30, 19].”