DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis

By Minfeng Zhu, Pingbo Pan, Yi Yang. 2019

672 citations recorded by the source corpus.

In this paper, we focus on generating realistic images from text descriptions. Current methods first generate an initial image with rough shape and color, and then refine the initial image to a high-resolution one. Most existing text-to-image synthesis methods have two main problems. (1) These methods depend heavily on the quality of the initial images. If the initial image is not well initialized, the following processes can hardly refine the image to a satisfactory quality. (2) Each word contributes a different level of importance when depicting different image contents, however, unchanged text representation is used in existing image refinement processes. In this paper, we propose the Dynamic Memory Generative Adversarial Network (DM-GAN) to generate high-quality images. The proposed method introduces a dynamic memory module to refine fuzzy image contents, when the initial images are not well generated. A memory writing gate is designed to select the important text information based on the initial image content, which enables our method to accurately generate images from the text description. We also utilize a response gate to adaptively fuse the information read from the memories and the image features. We evaluate the DM-GAN model on the Caltech-UCSD Birds 200 dataset and the Microsoft Common Objects in Context dataset. Experimental results demonstrate that our DM-GAN model performs favorably against the state-of-the-art approaches.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “We evaluate our model zero-shot by comparing it to three prior approaches: AttnGAN (Xu et al., 2018), DMGAN (Zhu et al., 2019), and DF-GAN (Tao et al., 2020), the last of which reports the best Inception Score (Salimans et al., 2016) and Fréchet Inception Distance (Heusel et al., 2017) on MS-COCO.”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “49 - - - - DM-GAN [72] 32 .”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “(Zhu et al., 2019) 32.64 DF-GAN (Tao et al., 2020) 21.42 DM-GAN + CL (Ye et al., 2021) 20.79 XMC-GAN (Zhang et al., 2021) 9.33 LAFITE (Zhou et al., 2021) 8.12 DALL-E (Ramesh et al., 2021) ∼ 28 LAFITE (Zhou et al., 2021) 26.94 GLIDE 12.24 GLIDE (Validation ﬁltered)”
Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “We compare L AFITE against a series of competitive systems: AttnGAN [51], Obj-GAN [23], DM-GAN [57], OP-GAN [13], DF-GAN [42] and XMC-GAN [54].”
CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “DM-GAN [55] and DF-GAN [45] introduced new architectures, e.g. dyanmic memory or deep fusion block, for better image reﬁnement.”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “We qualitatively compare the proposed method with several state-of-the-art methods, including some GAN-based methods [60, 61, 63, 67, 70, 71, 73], DALL-E [48] and CogView [13], on MSCOCO, CUB-200 and Oxford-102 datasets.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “We evaluate our approach over two popular base models, AttnGAN [42] and DM-GAN [48] on datasets CUB [37] and COCO [20].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “GAN-based models in particular have shown better sample quality [61, 64, 58, 66, 59, 26, 52, 42, 24].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “[86] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.”
Paint by Word (2021, background): “…generator that explicitly takes a language embedding as input [39]; the image quality of this approach can be improved using multi-stage generators [51, 52, 54], and semantic consistency can be improved through careful design of architectures and training losses [48, 58, 34, 49, 25, 43].”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, methodology): “Inception Score (IS) Following Zhu et al. (2019), we use all 30000 generated images.”
DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis (2021, methodology): “As shown in Figure 1, in the 1 column, compared with AttnGAN [3] and DM-GAN [5], only our proposed DAEGAN synthesizes the details of ‘wild animals’.”
Scaling up GANs for Text-to-Image Synthesis (2023, positive): “Lastly, we introduce a new, fast GAN-based up-sampler model that can improve the inference quality and speed of our method and diffusion models such as Ima-gen [59] and DALL · E 2 [53] (Section 3.4).”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “Moreover, to generate high-quality content, a multitude of strong-performing methods have been proposed, such as Transformer [82, 101, 17, 24], GANs [53, 7, 93, 110], VAEs [81, 67], Flow models [73, 6] and the current state-of-the-art diffusion models [31, 64, 57, 22, 68].”
Efficient Neural Architecture for Text-to-Image Synthesis (2020, methodology): “This architectural choice has been followed by most pre- 978-1-7281-6926-2/20/$31.00 ©2020 IEEE vious work, which have been adding small improvements, such as word-level features through Attention Mechanisms [16], Memory Networks [19], Siamese Networks [17] and a Mirror strategy [17].”
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers (2022, methodology): “Following [13, 66], we sample 30K images from MS COCO val2014 split and sample a caption for each image.”