Text-to-image Synthesis via Symmetrical Distillation Networks

By Mingkuan Yuan, Yuxin Peng. 2018

39 citations recorded by the source corpus.

Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task. The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps. The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions. For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g. VGG19), which can guide the training process of a new generative model on multiple levels to bridge the two gaps. The high-level representations can teach the generative model to extract necessary visual information from text descriptions, which can bridge the heterogeneous gap. The mid-level and low-level representations can lead it to learn structures and details of images respectively, which relieves the homogeneous gap. Therefore, we propose Symmetrical Distillation Networks (SDN) composed of a source discriminative model as "teacher" and a target generative model as "student". The target generative model has a symmetrical structure with the source discriminative model, in order to transfer hierarchical knowledge accessibly. Moreover, we decompose the training process into two stages with different distillation paradigms for promoting the performance of the target generative model. Experiments on two widely-used datasets are conducted to verify the effectiveness of our proposed SDN.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yuxin Peng
2. Yi Yang
3. Dacheng Tao
4. Yuandong Tian
5. Mingkuan Yuan
6. Xiaogang Wang
7. Luc Van Gool
8. Trevor Darrell

In Multimodal Learning

1. Mingkuan Yuan
2. Yuxin Peng
3. Minfeng Zhu
4. Pingbo Pan
5. Yuandong Tian
6. Yi Yang
7. Marcus Rohrbach
8. Dacheng Tao

Citations in the corpus

• DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, background): “[29] employed symmetrical distillation networks to minimize the multi-level difference between real and synthetic images.”
• RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge (2020, background): “Generating realistic images from text descriptions is one of the most active research areas in recent years [10, 12, 16, 21, 24, 26, 33, 37].”
• CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis (2020, methodology): “In our previous work [20], we only transfer the knowledge of image classification, which limits the per-”