SEAN: Image Synthesis With Semantic Region-Adaptive Normalization

By Peihao Zhu, Rameen Abdal, Yipeng Qin, Peter Wonka. 2019

502 citations recorded by the source corpus.

We propose semantic region-adaptive normalization (SEAN), a simple but effective building block for Generative Adversarial Networks conditioned on segmentation masks that describe the semantic regions in the desired output image. Using SEAN normalization, we can build a network architecture that can control the style of each semantic region individually, e.g., we can specify one style reference image per region. SEAN is better suited to encode, transfer, and synthesize style than the best previous method in terms of reconstruction quality, variability, and visual quality. We evaluate SEAN on multiple datasets and report better quantitative metrics (e.g. FID, PSNR) than the current state of the art. SEAN also pushes the frontier of interactive image editing. We can interactively edit images by changing segmentation masks or the style for any given region. We can also interpolate styles from two reference images per region.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “The StyleGAN-based architectures have been applied to numerous applications [38], [39], [40].”
Dynamic Neural Networks: A Survey (2021, neutral): “Sa & Sp Image segmentation [34], [129], [146], [148], [150], [154], [156], [216], [217], [218], [219], [220], Image-to-image translation [221], Object detection [110], [111], [147], [148], [164], Semantic image synthesis [222], [223], [224], Computer Image denoising [225], Fine-grained classificatio”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “Finally, there is the idea to combine the two techniques and rst use the encoder network to obtain an approximate embedding and then re ne it with an optimization algorithm (Zhu et al. 2020, 2016).”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “In this paper, we ﬁrst introduce a novel encoder architecture tasked with encoding an arbitrary image directly into W + .”
Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators? (2020, result): “tion of the generation result and thus is widely adopted in many studies [14, 19, 27, 29].”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “…tasks including image synthesis and editing (Portenier et al., 2018; Chen & Koltun, 2017; Dekel et al., 2018; Wang et al., 2018; Park et al., 2019; Zhu et al., 2020b; Jo & Park, 2019; Liu et al., 2021), inpainting (Pathak et al., 2016; Iizuka et al., 2017; Yang et al., 2017; Liu et al., 2018),…”
Generative Adversarial Transformers (2021, methodology): “Other works, such as SPADE [55, 73], employ region-based feature modulation for the task of layout-to-image translation, but, contrary to us, use ﬁxed segmentation maps and static class embeddings to control the visual features.”
Few-Shot Incremental Learning for Label-to-Image Translation (2022, background): “Existing works have made great progress in generation spatial alignment [36], diversity [45], fine details [47], and style controls [62].”
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (2021, background): “The first approach learns to control by providing the desired condition in training procedure and has shown remarkable performance on various tasks, such as segmentation mask conditioned generation [31, 59], style transfer [9, 50], and inpainting [23, 52].”
Image-to-Image Translation: Methods and Applications (2021, background): “SEAN [4] therefore designs semantic region-adaptive normalization layer to alleviate the two shortcomings.”
HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment (2020, methodology): “Inspired by SPADE [44] and SEAN [45] for semantic-guided image generation based on external parsing maps, our HiFaceGAN utilizes the SPADE layers to implement collaborative suppression and replenishment for multi-stage face renovation, which progressively replenishes plausible details based on hiera”
Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “To evaluate our method quantitatively, we used the pre-processed CelebAMask-HQ datasets [43], which contains face images and corresponding semantic masks (namely, 2,000 for test and 28,000 for training).”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, background): “…relationship: Meanwhile, [52] proposed an alternative non-Markovian noising process that has the same forward marginals as DDPM but has a distinct sampling process as follows: where, z ∼ N ( 0 , I ) and f θ ( x t , t ) is a the prediction of x 0 at t given x t and ϵ θ ( x t , t ) : This…”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “, label [20], [21], sketch [22], [23], text [24], [25], or control vector like gaze [26] or light direction [27].”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, background): “On the basis of unconditional generative models, conditional generative models target synthesizing images according to additional context such as image [4,17,23,30,41], segmentation mask [10,24,33,43], and text.”