Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis

By Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, Hongsheng Li. 2019

231 citations recorded by the source corpus.

Semantic image synthesis aims at generating photorealistic images from semantic layouts. Previous approaches with conditional generative adversarial networks (GAN) show state-of-the-art performance on this task, which either feed the semantic label maps as inputs to the generator, or use them to modulate the activations in normalization layers via affine transformations. We argue that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images. In order to better exploit the semantic layout for the image generator, we propose to predict convolutional kernels conditioned on the semantic label map to generate the intermediate feature maps from the noise maps and eventually generate the images. Moreover, we propose a feature pyramid semantics-embedding discriminator, which is more effective in enhancing fine details and semantic alignments between the generated images and the input semantic layouts than previous multi-scale discriminators. We achieve state-of-the-art results on both quantitative metrics and subjective evaluation on various semantic segmentation datasets, demonstrating the effectiveness of our approach.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, neutral): “Sa & Sp Image segmentation [34], [129], [146], [148], [150], [154], [156], [216], [217], [218], [219], [220], Image-to-image translation [221], Object detection [110], [111], [147], [148], [164], Semantic image synthesis [222], [223], [224], Computer Image denoising [225], Fine-grained classificatio”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “Since then, their work has been extended for many scenarios: high-resolution synthesis [38], unsupervised learning [27, 43, 22, 26], multi-modal image synthesis [44, 14, 6], and conditional image synthesis [30, 23, 28, 45, 5].”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “Mask-to-image translation is an area which has seen remarkable recent progress [17, 37, 27, 23].”
A Survey of Human-in-the-loop for Machine Learning (2021, background): “To tackle this challenge, many researchers build new datasets by generating samples, thereby speeding up model iteration and reducing the cost of data annotation [7, 8, 9, 10, 11].”
Few-Shot Incremental Learning for Label-to-Image Translation (2022, methodology): “Additionally, spatially-adaptive normalization [36], spatiallyvarying conditional convolution kernels [25] and classadaptive normalization [46] are proposed to avoid semantic information being washed away.”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, background): “…to allow image synthesis in various different settings: high-resolution (Wang et al., 2018a), semantic image (Park et al., 2019; Zhu et al., 2020b; Liu et al., 2019b), multi-domain (Choi et al., 2018), multimodal (Zhu et al., 2017b), and using a pre-trained generator (Nitzan et al., 2020;…”
BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models (2022, background): “Numerous problems in computer vision and graphics can be formulated as image-to-image translation problems, such as style transfer [3,9,13,22], semantic image synthesis [21,24,34,36,37, 40] and sketch-to-photo synthesis [2, 14, 43].”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “This task can be performed in the supervised setting [22, 26, 35, 49, 61, 72], where example pairs of corresponding images are available, as well as the unsupervised setting [14, 21, 32, 33, 35, 53, 71], where only two sets of images are available.”
Enhancing Photorealism Enhancement (2021, methodology): “feature space [8], [15], [71], processed via a separate network stream [72], or guide an auxiliary classifier [73].”
Hyper-LifelongGAN: Scalable Lifelong Learning for Image Conditioned Generation (2021, methodology): “For instance, for U-Net generator [17], only the decoder would be dynamic, parameterized as hypernetworks while the encoder remains deterministic [24]; for Resnet generator [19, 42], there would be a fixed sub-model while only last few layers are parameterized as hypernetworks [34].”
Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “ity [6, 22, 23, 32, 31], generate multi-modal outputs [23, 18, 44, 10], and simplify input annotations using bounding boxes [41, 30, 19].”
Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “GAN [15, 41, 46], on the other hand, is extensively applied in face generation [29, 30, 31], imageto-image translation [23, 74, 7, 26], style transfer [37, 21], and semantic image synthesis [60, 43, 38].”
You Only Need Adversarial Supervision for Semantic Image Synthesis (2020, methodology): “Since then, various generator and discriminator modifications have been introduced (Wang et al., 2018; Park et al., 2019; Liu et al., 2019; Tang et al., 2020c;b; Ntavelis et al., 2020).”
Dual Contrastive Loss and Attention for GANs (2021, background): “Significant improvements are obtained by careful architectural designs for generators [10, 98, 42, 43, 69], discriminators [82, 56] and new regularization techniques [58, 3, 25, 60, 101, 105, 96, 40, 106, 36].”