CogView: Mastering Text-to-Image Generation via Transformers

By Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang and 1 more. 2021

1,011 citations recorded by the source corpus.

Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Evaluation of text-conditional image synthesis on the MS-COCO [50] dataset: Our model outperforms autoregressive [17, 64] and GAN-based [105] methods by a significant margin when using 250 DDIM [81] steps .”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “Other works have adapted the VQ-VAE approach [52] to text-conditional image generation by training autoregressive transformers on sequences of text tokens followed by image tokens [40, 12, 1].”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Text-to-image generation [64, 72, 54, 65, 67, 45, 12, 41, 70] focuses on generating images from standalone text descriptions.”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, methodology): “e [29], CogView [11]) requires huge amounts of paired text-image data, which is too expensive to collect.”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “, DALL-E [48] and CogView [13], sequentially predict each image token depends on the text tokens as well as the previously predicted image tokens, i.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “In CogView (Ding et al., 2021), PB-Relax is proposed to remove bias terms and deduct extremum value in attention computation to avoid the problem, which unfortunately does not help avoid disconvergence in GLM-130B.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, background): “Model Dataset Batch size Base LR Weight decay DALL-E (Ramesh et al., 2021) and Cogview (Ding et al., 2021) also notice this issue and provide their solutions.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Most of the existing two-stage text-to-image generation models, including DALL-E [2], CogView [3] and Make-A-Scene [10], are decoder-only models.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “Moreover, efficient and effective task-specific fine-tuning is also an important research direction for the future application of PTMs (Soares et al., 2019; Ding et al., 2021b).”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “Unfortunately, the training is unstable for large models due to the overflow issue (characterized by NaN losses) [51].”
NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “Inspired by image/video generation [21, 31, 32, 33, 34] that uses VQ-VAE [20, 35, 36] to compress high-dimensional image into low-dimensional representations to ease the generation, we leverage VAE [22] to compress high-dimensional speech x into frame-level representations z (i.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Compared with DALLE [50], CogView [51] and NÜWA [52], whose sampling sizes are 512 , 60 and 60 , respectively, OFA outperforms these SOTA methods on FID and IS with a much smaller sampling size 24 .”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “Recently, image generation has got great success, one of the reasons is that they collect large-scale text-image data, e.g., CogView [25] collects more than 30 million high-quality (Chinese) text-image pairs.”
NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “I am so into summer, especially the sea; I hope I can go to the seaside to have some fun.”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, background): “…it is now possible to generate and modify high-quality images from natural language descriptions in a number of seconds (Ramesh et al., 2021; Ding et al., 2021; Nichol et al., 2021; Ramesh et al., 2022; Gafni et al., 2022; Yu et al., 2022; Saharia et al., 2022; Feng et al., 2022; Balaji et…”