CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis
By Mingkuan Yuan, Yuxin Peng. 2020
70 citations recorded by the source corpus.
Text-to-image synthesis (T2IS) has drawn increasing interest recently, which can automatically generate images conditioned on text descriptions. It is a highly challenging task that learns a mapping from a semantic space of text description to a complex RGB pixel space of image. The main issues of T2IS lie in two aspects: semantic consistency and visual quality. The distributions between text descriptions and image contents are inconsistent since they belong to different modalities. So it is ambitious to generate images containing consistent semantic contents with the text descriptions, which is the semantic consistency issue. Moreover, due to the discrepancy of data distributions between real and synthetic images in huge pixel space, it is hard to approximate the real data distribution for synthesizing photo-realistic images, which is the visual quality issue. For addressing the above issues, we propose a cross-task knowledge distillation (CKD) approach to transfer knowledge from multiple image semantic understanding tasks into T2IS task. There is amount of knowledge in image semantic understanding tasks to translate image contents into semantic representation, which is advantageous to address the issues of semantic consistency and visual quality for T2IS. Moreover, we design a multi-stage knowledge distillation paradigm to decompose the distillation process into multiple stages. By this paradigm, it is effective to approximate the distributions of real image and understand textual information for T2IS, which can improve the visual quality and semantic consistency of synthetic images. Comprehensive experiments on widely-used datasets show the effectiveness of our proposed CKD approach.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Learning with Recoverable Forgetting (2022, positive): “Besides, the multi-stage knowledge transfer is enabled by Yuan et al. [62] to de-sign a multi-stage knowledge distillation paradigm to decompose the distillation process.”
- DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “Due to its practical applications, text-to-image synthesis has become an active research area recently [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], M. Tao, S. Wu and F. Wu are with the School of Automation, Nanjing University of Posts and…”
- DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis (2020, background): “Due to its practical value, text-to-image synthesis has become an active research area recently [3,9,13,19–21,32,33,35,51,53,60].”
- More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, methodology): “To this end, we propose a novel knowledge distillation [15, 34, 63] approach to regularize the visual attention in captioner, by treating an image-text matching model as a weak supervision of grounding [19, 48].”
- CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “CKD [47] parses the image content by a hierarchical semantic representation to enhance the semantic consistency and visual quality of synthesized images.”
- CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2019, background): “nput text by learning two variables that are disentangled in the latent space. Text-SeGAN [5] focuses on devising a specic discriminator to regress the semantic relevance between text and image. CKD [47] parses the image content by a hierarchical semantic representation to enhance the semantic consi”