Scaling up GANs for Text-to-Image Synthesis

By Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, Taesung Park. Computer Vision and Pattern Recognition, 2023

693 citations recorded by the source corpus.

The recent success of text-to-image synthesis has taken the world by storm and captured the general public's imagination. From a technical standpoint, it also marked a drastic change in the favored architecture to design generative image models. GANs used to be the de facto choice, with techniques like StyleGAN. With DALL.E 2, autoregressive and diffusion models became the new standard for large-scale generative models overnight. This rapid shift raises a fundamental question: can we scale up GANs to benefit from large datasets like LAION? We find that naïvely increasing the capacity of the StyleGan architecture quickly becomes unstable. We introduce GigaGAN, a new GAN architecture that far exceeds this limit, demonstrating GANs as a viable option for text-to-image synthesis. GigaGAN offers three major advantages. First, it is orders of magnitude faster at inference time, taking only 0.13 seconds to synthesize a 512px image. Second, it can synthesize high-resolution images, for example, 16-megapixel images in 3.66 seconds. Finally, GigaGAN supports various latent space editing applications such as latent interpolation, style mixing, and vector arithmetic operations.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0930fe943d7d9b5bb943613d87c4ca92850dd43a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Analyzing and Improving the Training Dynamics of Diffusion Models (2023, positive): “Model averaging [27, 49, 57, 68, 72] is an indispensable technique in all high-quality image synthesis methods [2, 11, 23, 29, 31, 33, 46, 48, 54, 60, 63, 65].”
One-Step Diffusion with Distribution Matching Distillation (2023, background): “Among them, GANs have shown unprecedented quality in realism [4, 26– 28, 30, 67], particularly when the GAN loss can be combined with task-speciﬁc, auxiliary regression losses to mitigate training instability, ranging from paired image translation [24, 54, 79, 89] to unpaired image editing [37, 55,…”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, neutral): “5 (Rombach et al., 2022), GigaGAN (Kang et al., 2023), PixArt (Chen et al., 2024), Imagen (Saharia et al., 2022), RAPHAEL Xue et al. (2024), LlamaGen (Sun et al., 2024), SDXL (Podell et al., 2023), and SD3 (Esser et al., 2024); unified models including Gemini (Anil et al., 2023), NExT-GPT (Wu et al.”
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation (2024, background): “For instance, Imagen [41], GigaGAN [19] and Stable Diffusion [40] introduce an additional super-resolution network, while Stable Cascade [36] employs multiple diffusion networks to increase resolution progressively.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “2.1 3D G ENERATION Following the success of generative models on 2D images using GANs (Goodfellow et al., 2014; Karras et al., 2019; Gu et al., 2022; Kang et al., 2023) and diffusion models (Rombach et al., 2021; Podell et al., 2023; Ho et al., 2020; Song et al., 2021; Saharia et al., 2022), people…”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, neutral): “To conduct the human preference study, we use the Prolific platform (https://www.prolific.com).”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Recent advancements in text-to-image generation [4, 8, 17, 28], particularly diffusion models [13, 27, 31, 36, 37], have opened new frontiers in content creation.”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…78, 88, 79, 25, 79, 16, 28], referring object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and…”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “Modern text-to-image (T2I) generative models, such as DALL-E [7, 8], Imagen [9, 10], Stable Diffusion [5], StyleGAN-T [4], and GigaGAN [11], have demonstrated the remarkable ability to synthesize realistic, artistic, and detailed images based on textual descriptions.”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, neutral): “Additionally, GigaGAN (Kang et al., 2023)”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, neutral): “In fact, previous attempts at scaling GANs resulted in diminishing returns when scaling the generator [48, 24].”
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, background): “Meanwhile, new GAN and autoregressive models are still being developed [20, 44].”
Adversarial Diffusion Distillation (2023, methodology): “The main differences are extended training ( ∼ 2M iterations with a batch size of 2048, which is comparable to GigaGAN’s schedule [25]), the improved discriminator architecture proposed in Section 3.2, and R1 penalty applied at each discriminator head.”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “2 [30] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, background): “Modern visual synthesis systems are usually powered by large text-to-image foundation models [2, 4, 7, 8, 10, 16, 24, 27, 28].”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, positive): “First, there have been several approaches in generative adversarial network (GAN; Goodfellow et al. 2014) that try to accelerate training with better convergence using pretrained visual encoders (Sauer et al., 2021; Kumari et al., 2022; Sauer et al., 2022; 2023a; Kang et al., 2023).”