Controllable Text-to-Image Generation

By Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip H. S. Torr. 2019

424 citations recorded by the source corpus.

In this paper, we propose a novel controllable text-to-image generative adversarial network (ControlGAN), which can effectively synthesise high-quality images and also control parts of the image generation according to natural language descriptions. To achieve this, we introduce a word-level spatial and channel-wise attention-driven generator that can disentangle different visual attributes, and allow the model to focus on generating and manipulating subregions corresponding to the most relevant words. Also, a word-level discriminator is proposed to provide fine-grained supervisory feedback by correlating words with image regions, facilitating training an effective generator which is able to manipulate specific visual attributes without affecting the generation of other content. Furthermore, perceptual loss is adopted to reduce the randomness involved in the image generation, and to encourage the generator to manipulate specific attributes required in the modified text. Extensive experiments on benchmark datasets demonstrate that our method outperforms existing state of the art, and is able to effectively manipulate synthetic images using natural language descriptions.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “This lack of control conveys a notion of randomness and weak user-inﬂuence on the image content and context [34].”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Most of the recent works [45, 46, 42, 13, 23, 48, 43, 18, 22, 17, 1, 12, 6, 19, 33, 4] on the text-to-image synthesis problem are based on GANs.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “GAN-based models in particular have shown better sample quality [61, 64, 58, 66, 59, 26, 52, 42, 24].”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, methodology): “X-L XMERT even outperforms two specialized generation models, comparable to AttnGAN and ControlGAN.”
Text-Guided Neural Image Inpainting (2020, background): “More recent works [14, 16] exploit object and word-level information during synthesis through an attention mechanism.”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, background): “Early image generation methods conditioned on text description mainly based on GANs [26, 43, 44, 39, 14, 2, 46, 40, 42, 34], due to their powerful capability of high-fidelity image synthesis.”
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation (2023, background): “Deep generative models have achieved tremendous success on text-conditioned image generation [3, 6, 11, 12, 14, 20, 25, 29, 30, 32, 34, 35, 38, 40] and have recently attracted intensive attention.”
CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2019, methodology): “C SOA-I #Parameters Reed [34] 7.88 0.07 StackGAN [49] 8.45 0.03 996M StackGAN++ [50] 8.30 0.03 466M Lao [15] 8.94 0.20 Infer [11] 11.46 0.09 MirrorGAN [32] 26.47 0.41 SEGAN [39] 27.86 0.31 ControlGAN [16] 24.06 0.60 SD-GAN [44] 35.69 0.50 DMGAN [52] 30.49 0.57 88.56% 33.44% 48.03% 223M AttnGAN [43] ”
CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis (2020, methodology): “To improve the semantic consistency between the generated image and the input text, ControlGAN [16] applies the matching scheme of DAMSM in AttnGAN [43] in all 3-level discriminators.”
Benchmark for Compositional Text-to-Image Synthesis (2021, methodology): “ControlGAN[19] aims at the user-controllable text-to-5 image synthesis by a word-level spatial and channel-wise attention-driven generator, a word-level discriminator, and a perceptual loss.”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, methodology): “Various methods have been studied leading to significant improvements in editing quality and correctness [55, 60, 51, 27, 11, 61, 33, 35, 17, 28, 7, 23, 40].”
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis (2020, background): “Due to its practical applications, text-to-image synthesis has become an active research area recently [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], M. Tao, S. Wu and F. Wu are with the School of Automation, Nanjing University of Posts and…”
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis (2020, result): “Although impressive results have been presented by previous works [9,19,21,32,33,51,60], there still remain three problems.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “Particularly, text-to-image synthesis [52, 20, 50, 58, 29, 51, 26] learns a mapping from textual descriptions to images.”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “Text-to-Image Synthesis There are mainly three types of models for text-to-image synthesis: GANbased (Tao et al., 2022; Zhu et al., 2019; Li et al., 2019; Fu et al., 2020; El-Nouby et al., 2019), autoregressive (Gu et al., 2022a; Lee et al., 2022; Ding et al., 2022) and diffusion models (Liu et…”
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, methodology): “The text-to-image synthesis task [66, 67, 30, 63, 72, 55] further employs natural language descriptions as controls.”