A Learned Representation For Artistic Style

By Vincent Dumoulin, Jonathon Shlens, Manjunath Kudlur. 2016

1,271 citations recorded by the source corpus.

The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “In addition, we adopt instance normalization in our score network, inspired by its superior performance in some image generation tasks [57, 13, 23], and we”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “Learned affine transformations then specialize w to styles y = (ys,yb) that control adaptive instance normalization (AdaIN) [26, 16, 20, 15] operations after each convolution layer of the synthesis network g.”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “We provide class information to G with class-conditional BatchNorm (Dumoulin et al., 2017; de Vries et al., 2017) and to D with projection (Miyato & Koyama, 2018).”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “For the generator network of conditional GANs, we used conditional batch normalization (CBN) (Dumoulin et al., 2017; de Vries et al., 2017).”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “This often takes the form of class-conditional normalization statistics [14, 10] as well as discriminators with heads that are explicitly designed to behave like classifiers p(y|x) [34].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “Modulation approaches emerge from the style transfer literature (Dumoulin et al., 2016; Huang & Belongie, 2017) and are well exploited in state-of-the-art unconditional or class-conditional GANs.”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “In [5], [15], [42], [43], conditional information is provided to the generator by conditional gains and bias in the Batch Normalization [29] layers.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “In contrast to [5] and in alignment with [6], we find it beneficial not to use a hierarchical latent space, but to directly feed the same input vector z to BatchNorm at every layer in the generator.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “1Conditional normalization [11, 19] uses external data to denormalize the normalized activations; i.”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, methodology): “(2021) for IC-GAN which rely on conditional batch normalization (Dumoulin et al., 2017).”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “FiLM can be thought of as a generalization of Conditional Normalization, which has proven highly successful for image stylization (Dumoulin, Shlens, and Kudlur 2017; Ghiasi et al. 2017; Huang and Belongie 2017), speech recognition (Kim, Song, and Bengio 2017), and visual question answering (de…”
cGANs with Projection Discriminator (2018, methodology): “The method with auxiliary classiﬁer loss seems effective in improving the visual appearance, but not in training faithful generative model. et al., 2017b; de Vries et al., 2017) was used in place of the standard batch normalization layer.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “There are some recent works addressing this problem, but they are either still limited to a finite set of styles [11, 32, 55, 5], or much slower than the single-style transfer methods [6].”
Modulating early visual processing by language (2017, methodology): “In CIN, all parameters, including the transposed convolutional ﬁlters, are trained.”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “Inspired by recent works that use aﬃne transformation parameters in normalization layers to represent styles [54, 72–74], we equip the residual blocks with Adaptive Instance Normalization (AdaIN) [54] layers whose parameters are dynamically generated by a multilayer perceptron (MLP) from the style…”
Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, methodology): “Specifically, there are 3 interpolation-based upsampling operations, including area interpolation, bilinear interpolation [7], nearest-neighbor interpolation [21].”
TADAM: Task dependent adaptive metric for improved few-shot learning (2018, methodology): “As for task conditioning, [3, 18, 19] proposed conditional batch normalization for style transfer and visual reasoning.”
Bayesian Hypernetworks (2017, methodology): “Conditional Batch Norm (CBN) (Dumoulin et al., 2016), and the closely related Conditional Instance Normalization (CIN) (Huang and Belongie, 2017; Ulyanov et al., 2016), and Feature-wise Linear Modulation (FiLM) (Perez et al., 2017; Kirkpatrick et al., 2016) methods can be viewed as speciﬁc forms of…”
GAN Memory with No Forgetting (2020, methodology): “Moving well beyond the style-transfer literature modulating image features to manipulate its style [18, 30, 17], we discover that one can even modulate the “style” of a source generative/discriminative process (e.”