Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

By Xun Huang, Serge J. Belongie. 2017

5,355 citations recorded by the source corpus.

Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “We use a stylized version of the ImageNet test set [34, 44].”
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, methodology): “…a new data set (termed Stylized-ImageNet or SIN) by strip-ping every single image of its original texture and replacing it with the style of a randomly selected painting through AdaIN style transfer (Huang & Belongie, 2017) (see examples in Figure 3) with a stylization coefﬁcient of α = 1 .”
Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Affine transforms then produce styles that control the layers of the synthesis network g via adaptive instance normalization (AdaIN) [18, 8, 11, 7].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “In addition, we adopt instance normalization in our score network, inspired by its superior performance in some image generation tasks [57, 13, 23], and we”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “Learned affine transformations then specialize w to styles y = (ys,yb) that control adaptive instance normalization (AdaIN) [26, 16, 20, 15] operations after each convolution layer of the synthesis network g.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “…layers used in the modulation blocks include batch normalization (Chen et al., 2019; Odena et al., 2018), adaptive instance normalization (Huang & Belongie, 2017; Karras et al., 2019a), and weight demodulation (Karras et al., 2019b) referred to the weight normalization (Salimans &…”
Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “16 pixels improves precision (E), whereas disabling instance normalization in the AdaIN operation [11], unexpectedly, improves recall (F).”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Conditional normalization layers include the Conditional Batch Normalization (Conditional BatchNorm) [11] and Adaptive Instance Normalization (AdaIN) [19].”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “2017) and Adaptive Instance Norm (Huang and Belongie 2017) for image stylization, Dynamic Layer Norm for speech recognition (Kim, Song, and Bengio 2017), and Conditional Batch Norm for general visual question answering on complex scenes such as VQA and GuessWhat?! (de Vries et al.”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “We do not use IN layers in the style encoder, since IN removes the original feature mean and variance that represent important style information [54].”
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, methodology): “We apply the fast style transfer method AdaIN [Huang and Belongie, 2017] with hyperparameter α = 1 to the training data, replacing the original texture with the randomly chosen texture information of Kaggle’s Painter by Numbers 5 dataset.”
Bayesian Hypernetworks (2017, methodology): “Conditional Batch Norm (CBN) (Dumoulin et al., 2016), and the closely related Conditional Instance Normalization (CIN) (Huang and Belongie, 2017; Ulyanov et al., 2016), and Feature-wise Linear Modulation (FiLM) (Perez et al., 2017; Kirkpatrick et al., 2016) methods can be viewed as speciﬁc forms of…”
GAN Memory with No Forgetting (2020, methodology): “FiLM imposes simple element-wise affine transformations to latent features of a neural network, showing remarkable effectiveness in various domains [30, 17, 33, 58].”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “rated into the decoder and duration and prosody predictors using adaptive instance normalization (AdaIN) [44], allowing the model to generate speech with varying duration, prosody, and emotions.”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “In contrast, we have adapted Vocos to be a conditional GAN with a projection discriminator (Miyato & Koyama, 2018), and have incorporated adaptive layer normalization (Huang & Belongie, 2017) into the generator.”
StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “We use adaptive instance normalization (AdaIN) [12, 18] to inject s into G.”
Context Encoding for Semantic Segmentation (2018, methodology): “Recent work in style transfer manipulates the featuremap mean and variance [11, 22] or second order statistics to enable in-network style switch [55].”