RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge

By Jun Cheng, Fuxiang Wu, Yuandong Tian, Lei Wang, Dapeng Tao. 2020

124 citations recorded by the source corpus.

Text-to-image synthesis is a challenging task that generates realistic images from a textual sequence, which usually contains limited information compared with the corresponding image and so is ambiguous and abstractive. The limited textual information only describes a scene partly, which will complicate the generation with complementing the other details implicitly and lead to low-quality images. To address this problem, we propose a novel rich feature generating text-to-image synthesis, called RiFeGAN, to enrich the given description. In order to provide additional visual details and avoid conflicting, RiFeGAN exploits an attention-based caption matching model to select and refine the compatible candidate captions from prior knowledge. Given enriched captions, RiFeGAN uses self-attentional embedding mixtures to extract features across them effectually and handle the diverging features further. Then it exploits multi-captions attentional generative adversarial networks to synthesize images from those features. The experiments conducted on widely-used datasets show that the models can generate images from enriched captions effectually and improve the results significantly.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
Cycle-Consistent Inverse GAN for Text-to-Image Synthesis (2021, methodology): “StackGAN++ [37] 4.04 ± 0.06 - AttnGAN [35] 4.36 ± 0.03 23.98 MirrorGAN [20] 4.56 ± 0.05 - ControlGAN [15] 4.58 ± 0.09 - SD-GAN [36] 4.67 ± 0.09 - DM-GAN [42] 4.75 ± 0.07 16.09 DF-GAN [26] 4.86 ± 0.04 19.24 RiFeGAN [6] 5.23 ± 0.09 - DALL-E (zero-shot) [21] 2.85 ± 0.04 51.00 CI-GAN (Ours) 5.72 ± 0.12 ”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “…+ dialogue [93], [95], [99] caption + layout [104], [97], [108], [103] caption + semantic masks [109], [110], [113], [114], [115], [38] scene graphs [116], [121], [122], [124], [125] multiple captions [91], [92] multiple captions + mouse traces [126] Table 2: Methods grouped by their supervision.”
DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis (2021, methodology): “RiFeGAN is particularly designed for datasets with fine-grained visual details, e.g., CUB-200.”
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis (2020, background): “Due to its practical value, text-to-image synthesis has become an active research area recently [3,9,13,19–21,32,33,35,51,53,60].”
TediGAN: Text-Guided Diverse Image Generation and Manipulation (2020, methodology): “Furthermore, the pretrained text-image matching model they used fails to exploit attribute-level cross-modal information and leads to mismatched attributes when generating images from texts [52, 26, 62, 6], or undesired changes of irrelevant”
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “[18].”
TediGAN: Text-Guided Diverse Face Image Generation and Manipulation (2021, methodology): “Furthermore, the pretrained text-image matching model they used fails to exploit attribute-level cross-modal information and leads to mismatched attributes when generating images from texts [39, 19, 47, 5], or undesired changes of irrelevant attributes when manipulating images [8, 26, 20, 21].”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “Other approaches explored introducing memory mechanisms [63, 24], cycle-consistency constraint [37, 17], siamese networks [57, 47], and prior knowledge [36, 4] to improve the text-to-image synthesis performance.”