CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

By Ming Ding, Wendi Zheng, Wenyi Hong, Jie Tang. 2022

425 citations recorded by the source corpus.

The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel auto-regressive generation. We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, Cross-modal general language model (CogLM), and finetune it for fast super-resolution. The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Yi Yang
6. Trevor Darrell
7. Chen Change Loy
8. Andrew Zisserman

In Generative Models

1. Jun-Yan Zhu
2. Richard Zhang
3. Eli Shechtman
4. Alexei A. Efros
5. Ying Shan
6. Xintao Wang
7. Jia-Bin Huang
8. Yi Ren

In Multimodal Learning

1. Ying Shan
2. Devi Parikh
3. Lei Zhang
4. Hongsheng Li
5. William Yang Wang
6. Jie Tang
7. Ming Ding
8. Zhe Gan

In Model Architectures

1. Jiashi Feng
2. Jinglin Liu
3. Hongsheng Li
4. Yi Ren
5. Rongjie Huang
6. Zhou Zhao
7. Daquan Zhou
8. Jie Tang

Citations in the corpus

• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “CogVideo (Hong et al., 2022) is built on top of a CogView2 (Ding et al., 2022) T2I model with a multi-frame-rate hierarchical training strategy.”
• IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, methodology): “Early works, such as DALLE [13], CogView [14, 15] and Make-A-Scene [16], are autoregressive models.”
• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting (2022, neutral): “There has been much recent work on text-guided image inpainting [1, 3, 5, 7, 19, 27, 31, 33].”
• Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, background): “CogVideo [14] improves video generation quality by extending CogView-2 [5] to T2V generation through the incorporation of temporal attention modules and pre-trained text-to-image models.”
• Video-P2P: Video Editing with Cross-Attention Control (2023, positive): “DALL-E [32] first considers the text-to-image (T2I) generation task as a sequence-to-sequence translation problem, with subsequent research improving generation quality [6, 10, 50].”
• PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents (2023, background): “…literature, development of foundational models has been the main driving force in artificial intelligence, for example, large language models [26,8,2,23] trained with either autoregressive prediction or masked token inpainting, and computer vision models [17,25,30] trained by contrasting…”
• Inversion-based Style Transfer with Diffusion Models (2022, background): “Text-to-image synthesis Text-guided synthesis methods can also be used to generate artistic images [11, 40, 41, 45, 56].”
• CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (2022, methodology): “We build CogVideo based on a pretrained text-to-image model, CogView2 [6], in order to inherit the knowledge learned from the text-image pretraining.”
• ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation (2023, background): “Deep generative models have achieved tremendous success on text-conditioned image generation [3, 6, 11, 12, 14, 20, 25, 29, 30, 32, 34, 35, 38, 40] and have recently attracted intensive attention.”
• InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, neutral): “52, 53, 54], GAN inversion [55, 56], GAN-based approaches [57], and diffusion models [8, 9, 58, 59].”
• InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “We encode X into a global concept embedding c g via Concept Encoder E g and obtain”
• ControlVideo: Training-free Controllable Text-to-Video Generation (2023, background): “Through pre-training on billions of image-text pairs, large-scale generative models [1, 2, 3, 4, 14, 22, 26, 27, 29, 32, 33, 42] have made remarkable progress in creative and photo-realistic image generation.”
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, methodology): “Recent large text-to-image models such as Imagen [58], DALL-E2 [51], Parti [69], CogView2 [17] and Stable Diffusion [55] demonstrated unprecedented semantic generation.”
• Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “Second, we fine-tune the T2I model for video generation, gaining the advantage of adapting the model weights effectively, compared to freezing the weights as in CogVideo (Hong et al., 2022).”