StyleSwin: Transformer-based GAN for High-resolution Image Generation

By Bo Zhang, Shuyang Gu, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, Baining Guo. Computer Vision and Pattern Recognition, 2021

311 citations recorded by the source corpus.

Despite the tantalizing success in a broad of vision tasks, transformers have not yet demonstrated on-par ability as ConvNets in high-resolution image generative modeling. In this paper, we seek to explore using pure transformers to build a generative adversarial network for high-resolution image synthesis. To this end, we believe that local attention is crucial to strike the balance between computational efficiency and modeling capacity. Hence, the proposed generator adopts Swin transformer in a style-based architecture. To achieve a larger receptive field, we propose double attention which simultaneously leverages the context of the local and the shifted windows, leading to improved generation quality. Moreover, we show that offering the knowledge of the absolute position that has been lost in window-based transformers greatly benefits the generation quality. The proposed StyleSwin is scalable to high resolutions, with both the coarse geometry and fine structures benefit from the strong expressivity of transformers. However, blocking artifacts occur during high-resolution synthesis because performing the local attention in a block-wise manner may break the spatial coherency. To solve this, we empirically investigate various solutions, among which we find that employing a wavelet discriminator to examine the spectral discrepancy effectively suppresses the artifacts. Extensive experiments show the superiority over prior transformer-based GANs, especially on high resolutions, e.g., $1024 \times$ 1024. The StyleSwin, without complex training strategies, excels over StyleGAN on CelebA-HQ 1024, and achieves on-par performance on FFHQ-1024, proving the promise of using transformers for high-resolution image generation. The code and pretrained models are available at https://github.com/microsoft/StyleSwin.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5553f9508dd1056ecc20c5b1f367e9a07e2c7e81

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “A steadily-developed line of work [2, 5, 48, 55,57] carefully dissects the GAN’s latent space, aiming to ﬁnd semantic disentangled latent factors.”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Multiple classes of generative models have utilized Transformers [73] as back-bone, such as, Generative adversarial networks [42, 47, 85], autoregressive [10, 11, 20, 21, 27, 59, 74, 77, 78, 80, 81] and diffusion [2, 22, 41, 50, 53, 87] models.”
Pretraining is All You Need for Image-to-Image Translation (2022, methodology): “Rather than using GANs [14, 3, 25, 57] that mainly work for speciﬁc domains ( e.g. , faces), we opt to use the diffusion model [19, 47, 11] which emerges to show impressive expressivity of synthesizing a wide variety of images.”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, methodology): “We tested recent unconditional image generation models - StyleSwin [9] and Latent Diffusion Models (LDM) [7] on both few-shot and lifelong few-shot image generation tasks.”
Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (2023, background): “…in many computer vision tasks and achieved impressive results in high-level vision ( e.g. , image classification [2, 4, 14], semantic segmentation [7, 55, 67], object detection [3, 13, 62], etc. ) and low-level vision ( e.g. , image restoration [6, 11, 60], image synthesis [20, 21, 64], etc. ).”
Dilated Neighborhood Attention Transformer (2022, background): “It also inspired other works that extended it to different tasks beyond the ones explored in the paper, such as generation [48], restoration [22], masked image modeling [45], video action recognition [28], and more.”
NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “I am so into summer, especially the sea; I hope I can go to the seaside to have some fun.”
LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, unknown): “…the untapped potential of Transformers in visual tasks, sparking a surge of research aimed at refining their architecture design [45] and training methodologies [18, 31, 46, 47, 53], boosting computational efficiency [3,22,33,49], and extending their application scope [8, 25, 26, 38, 44, 56, 58].”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “The state of generative modeling [5,14,15,28,50,65,75] has seen rapid progress in past years.”