Semantic Image Synthesis With Spatially-Adaptive Normalization
By Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu. 2019
3,180 citations recorded by the source corpus.
We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the network, forcing the network to memorize the information throughout all the layers. Instead, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned affine transformation. Experiments on several challenging datasets demonstrate the superiority of our method compared to existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows users to easily control the style and content of image synthesis results as well as create multi-modal results. Code is available upon publication.
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Eli Shechtman
- Ben Mildenhall
- Alexei A. Efros
- Daniel Cohen-Or
- Jun-Yan Zhu
- Pratul P. Srinivasan
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Daniel Cohen-Or
- Alexei A. Efros
- Stefano Ermon
- Bolei Zhou
- Ming-Hsuan Yang
Citations in the corpus
- CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “GauGAN [34] We take officially released GauGAN model pretrained on COCO [27], and generate real and fake image pairs.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Contrary to the common use of segmentation for explicit conditioning as employed in many GAN-based methods [24, 62, 42], our segmentation tokens provide implicit conditioning in the sense that the generated image and image tokens are not constrained to use the segmentation information, as there is…”
- Generative Adversarial Networks (2021, neutral): “The Training Process involves iteratively updating the parameters of the Generator Network and Discriminator Network to improve their performance.”
- Projected GANs Converge Faster (2021, background): “Providing multi-scale feedback with one or multiple discriminators has been helpful for both image synthesis [23, 24] and image-to-image translation [43, 64].”
- High-Resolution Complex Scene Synthesis with Transformers (2021, background): “in class-conditional [4, 24], text-conditional [23, 34] or pixelwise conditional settings [20, 33, 30].”
- On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “A wide range of image and video synthesis applications [45, 54, 67, 83] have been enabled, as a result of tremendous progress in deep generative models such as GANs [9, 28, 34, 37, 59], VAEs [41, 55, 61], autoregressive models [52], flow-based models [18, 40], and energy-based models [21, 50, 63].”
- Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “…also extend the application of modulation approaches to image-conditional GANs with the style representations fully determined by an input image (Park et al., 2019; Huang et al., 2018; Liu et al., 2019b); however, the absence of stochasticity makes them hardly generalizable to the settings…”
- Taming Transformers for High-Resolution Image Synthesis (2020, background): “This inductive bias towards local interactions thus leads to efficient computations, but the wide range of specialized layers which are introduced into CNNs to handle different synthesis tasks [46, 70, 59, 74, 73] suggest that this bias is often too restrictive.”
- WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “The structure is inspired by spatially-adaptive denormalization (Park et al., 2019).”
- Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “In [5], [15], [42], [43], conditional information is provided to the generator by conditional gains and bias in the Batch Normalization [29] layers.”
- Object-Centric Image Generation from Layouts (2020, methodology): “In addition, some methods also consider multi-modal data, such as instance segmentation alongside pixel-wise semantic segmentation masks (Park et al. 2019; Wang et al. 2018).”
- StyleGAN-NADA (2021, methodology): “In addition to StyleGAN, we investigated our model’s ability to convert existing classes in a more localized manner using OASIS [43], a SPADE-like [36] model that synthesizes images from segmentation masks.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “Over the past couple of years, GANs [14] were the state-of-the-art, with their latent space and conditional inputs being well-studied for controllable manipulation [48, 60] and generation [27, 29, 47, 82].”
- Rewriting a Deep Generative Model (2020, methodology): “…and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo stylization [25,38,56,54], image-to-image translation [35,86,9,73,55,32], and face editing and…”
- Dynamic Neural Networks: A Survey (2021, neutral): “Sa & Sp Image segmentation [34], [129], [146], [148], [150], [154], [156], [216], [217], [218], [219], [220], Image-to-image translation [221], Object detection [110], [111], [147], [148], [164], Semantic image synthesis [222], [223], [224], Computer Image denoising [225], Fine-grained classificatio”
- Self-Attention Generative Adversarial Networks (2018, background): “GANs have achieved great success in various image generation tasks, including image-to-image translation (Isola et al., 2017; Zhu et al., 2017; Taigman et al., 2017; Liu & Tuzel, 2016; Xue et al., 2018; Park et al., 2019), image super-resolution (Ledig et al.”
- GAN Memory with No Forgetting (2020, background): “Style-transfer techniques An extensive literature [23, 71, 79, 13, 62, 38] has explored how one can manipulate the style of an image (e.”