StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

By Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, Timo Aila. International Conference on Machine Learning, 2023

300 citations recorded by the source corpus.

Text-to-image synthesis has recently seen significant progress thanks to large pretrained language models, large-scale training data, and the introduction of scalable model families such as diffusion and autoregressive models. However, the best-performing models require iterative evaluation to generate a single sample. In contrast, generative adversarial networks (GANs) only need a single forward pass. They are thus much faster, but they currently remain far behind the state-of-the-art in large-scale text-to-image synthesis. This paper aims to identify the necessary steps to regain competitiveness. Our proposed model, StyleGAN-T, addresses the specific requirements of large-scale text-to-image synthesis, such as large capacity, stable training on diverse datasets, strong text alignment, and controllable variation vs. text alignment tradeoff. StyleGAN-T significantly improves over previous GANs and outperforms distilled diffusion models - the previous state-of-the-art in fast text-to-image synthesis - in terms of sample quality and speed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d33be7c99b7265652edbb2b0f832b479e2162ae1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “…transformers akin to GPT-2 and VQ-GAN [49, 19], with the only modification of substituting traditional layer normalization for adaptive normalization (AdaLN) – a choice motivated by its widespread adoption and proven effectiveness in visual generative models [33, 34, 32, 57, 56, 29, 46, 12].”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “09 second, on par with one of the state-of-the-art GANs, StyleGAN-T [4] .”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “1, 2 [58] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.”
One-Step Diffusion with Distribution Matching Distillation (2023, methodology): “Among them, GANs have shown unprecedented quality in realism [4, 26– 28, 30, 67], particularly when the GAN loss can be combined with task-speciﬁc, auxiliary regression losses to mitigate training instability, ranging from paired image translation [24, 54, 79, 89] to unpaired image editing [37, 55,…”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, background): “Through pre-training on billions of image-text pairs, large-scale generative models [1, 2, 3, 4, 14, 22, 26, 27, 29, 32, 33, 42] have made remarkable progress in creative and photo-realistic image generation.”
Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis (2023, unknown): “These include autoregressive models [22, 2], GANs [27], and diffusion models [23, 21, 24].”
Adversarial Diffusion Distillation (2023, methodology): “For the feature network F , Sauer et al. [59] ﬁnd vision transformers (ViTs) [9] to work well, and we ablate different choice for the ViTs objective and model size in Section 4.”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, unknown): “However, in one-shot scenarios, CFG simply oversaturates samples rather than improving text-alignment [48].”
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation (2023, background): “Deep generative models have achieved tremendous success on text-conditioned image generation [3, 6, 11, 12, 14, 20, 25, 29, 30, 32, 34, 35, 38, 40] and have recently attracted intensive attention.”
Scaling up GANs for Text-to-Image Synthesis (2023, unknown): “Concurrent with our method, StyleGAN-T [61] and GALIP [66] share similar goals and make complementary insights to ours.”