Instance Normalization: The Missing Ingredient for Fast Stylization

By Dmitry Ulyanov, Andrea Vedaldi, Victor S. Lempitsky. 2016

4,163 citations recorded by the source corpus.

It this paper we revisit the fast stylization method introduced in Ulyanov et. al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images. The change is limited to swapping batch normalization with instance normalization, and to apply the latter both at training and testing times. The resulting method can be used to train high-performance architectures for real-time image generation. The code will is made available on github at this https URL. Full paper can be found at arXiv:1701.02096.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking ImageNet Pre-Training (2018, neutral): “Successful forms of normalization include normalized parameter initialization [11, 16] and activation normalization layers [20, 2, 46, 48].”
Weight Standardization (2019, methodology): “To explore other dimensions [2, 33], Layer Normalization [3] normalizes data on the channel dimension, Instance Normalization [40] performs BN for each sample individually, and Group Nor-malization [42] ﬁnds a better middle point between Layer Normalization and Instance Normalization.”
Group Normalization (2018, methodology): “There have been existing methods, such as Layer Normalization (LN) [3] and Instance Normalization (IN) [61] (Figure 2), that also avoid normalizing along the batch dimension.”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “In addition, we adopt instance normalization in our score network, inspired by its superior performance in some image generation tasks [57, 13, 23], and we”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Specifically, while normalization lay ers such as the InstanceNorm [43] are essential pieces in almost all the state-of-the-art conditional image synthesis models [45], they tend to wash away semantic information when applied to uniform or flat segmentation masks.”
Improved Training of Wasserstein GANs (2017, background): “, 2016), weight normalization (Salimans & Kingma, 2016) and instance normalization (Ulyanov et al., 2016).”
Image-to-Image Translation with Conditional Adversarial Networks (2016, positive): “This approach to batch normalization, when the batch size is set to 1, has been termed “instance normalization” and has been demonstrated to be effective at image generation tasks [51].”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “We use instance normalization [28] for the generator but no normalization for the discriminator.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, positive): “[21], we use instance normalization [49].”
A Learned Representation For Artistic Style (2016, methodology): “Building off the instance normalization technique proposed in Ulyanov et al. (2016b), we augment the γ and β parameters so that they’re N × C matrices, where N is the number of styles being modeled and C is the number of output feature maps.”
Fixup Initialization: Residual Learning Without Normalization (2019, methodology): “Besides image classiﬁcation, various normalization techniques (Ulyanov et al., 2016; Ba et al., 2016; Salimans & Kingma, 2016; Wu & He, 2018) have been found essential to achieving good performance on other tasks, such as machine translation (Vaswani et al., 2017) and generative modeling (Zhu et…”
Fully Convolutional Speech Recognition (2018, background): “The front-end finally applies a log-compression and a per-channel mean-variance normalization (equivalent to an instance normalization layer [19]).”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “At each step t , we encode a box tensor B t by a series of downsampling layers, where each downsampling layer is implemented by a stride-2 convolution followed by instance-wise normalization [29] and ReLU.”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “For the normalization layers, we adopt the instance normalization (Ulyanov et al., 2016) that is widely used in image translation (Zhu et al., 2017) and style transfer (Johnson et al., 2016).”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, background): “• N stands for the normalization type of this block, including three options: batch normalization[21], instance normalization[55], and no normalization.”