Exploring the structure of a real-time, arbitrary neural artistic stylization network

By Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens. 2017

312 citations recorded by the source corpus.

In this paper, we present a method which combines the flexibility of the neural algorithm of artistic style with the speed of fast style transfer networks to allow real-time stylization using any content/style image pair. We build upon recent work leveraging conditional instance normalization for multi-style transfer networks by learning to predict the conditional instance normalization parameters directly from a style image. The model is successfully trained on a corpus of roughly 80,000 paintings and is able to generalize to paintings previously unobserved. We demonstrate that the learned embedding space is smooth and contains a rich structure and organizes semantic information associated with paintings in an entirely unsupervised manner.

Subfields: Cognitive Neuroscience, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Analyzing and Improving the Image Quality of StyleGAN (2019, positive): “Affine transforms then produce styles that control the layers of the synthesis network g via adaptive instance normalization (AdaIN) [21, 9, 13, 8].”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “Learned affine transformations then specialize w to styles y = (ys,yb) that control adaptive instance normalization (AdaIN) [26, 16, 20, 15] operations after each convolution layer of the synthesis network g.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “Various forms of CN have proven highly effective across a number of domains: Conditional Instance Norm (Dumoulin, Shlens, and Kudlur 2017; Ghiasi et al. 2017) and Adaptive Instance Norm (Huang and Belongie 2017) for image stylization, Dynamic Layer Norm for speech recognition (Kim, Song, and Bengio…”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “Inspired by recent works that use aﬃne transformation parameters in normalization layers to represent styles [54, 72–74], we equip the residual blocks with Adaptive Instance Normalization (AdaIN) [54] layers whose parameters are dynamically generated by a multilayer perceptron (MLP) from the style…”
GAN Memory with No Forgetting (2020, background): “Style-transfer techniques An extensive literature [23, 71, 79, 13, 62, 38] has explored how one can manipulate the style of an image (e.”
Learning Visual Reasoning Without Strong Priors (2017, methodology): “Our model processes the multi-modal question-image input using a RNN and CNN combined via Conditional Batch Normalization (CBN).”
Universal Style Transfer via Feature Transforms (2017, background): “or each style [6]. To achieve arbitrary style transfer, Chen et al. [3] ﬁrst propose to swap the content feature with the closest style feature locally. Meanwhile, inspired by [6], two following work [30, 11] turn to learn a general mapping from the style image to style parameters. One closest relat”
AdaptIS: Adaptive Instance Selection Network (2019, methodology): “But how can we generate an object mask based on this ”characteristic” vector? Recently an elegant solution for similar problems was proposed in the literature on style transfer [12, 7, 10] and image generation [13].”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modiﬁes color distributions and texture patterns of content images.”
Rethinking and Improving the Robustness of Image Style Transfer (2021, background): “A consistent observation from this work [43, 24, 30, 23, 25, 19, 9, 2, 8] is that the correlation between the activations of a pre-trained VGG [33] network has remarkable ability to capture the visual style of an image.”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, background): “Similar trade-off between flexibility and capability also exists in [5, 22, 15, 24, 10], where all local feature points of the content image are processed by the same transformation function based on style images.”
A Closed-form Solution to Photorealistic Image Stylization (2018, methodology): “A number of methods have been developed [14, 34, 11, 16, 3, 8, 9, 17] to further improve its stylization performance and speed.”
Neural Style Transfer: A Review (2017, background): “’s algorithm [55] is data-driven and their stylisation quality is very dependent on the varieties of training styles.”
Neural Style Transfer via Meta Networks (2018, background): “Another concurrent works by [10] propose learning to predict the conditional instance normalization parameters from a style image.”
Shape-Texture Debiased Neural Network Training (2020, background): “Later follow-ups further improve the generation quality and speed (Huang & Belongie, 2017; Chen & Schmidt, 2016; Ghiasi et al., 2017; Li et al., 2017).”