Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis

By Dmitry Ulyanov, Andrea Vedaldi, Victor S. Lempitsky. 2017

866 citations recorded by the source corpus.

The recent work of Gatys et al., who characterized the style of an image by the statistics of convolutional neural network filters, ignited a renewed interest in the texture generation and image stylization problems. While their image generation technique uses a slow optimization process, recently several authors have proposed to learn generator neural networks that can produce similar outputs in one quick forward pass. While generator networks are promising, they are still inferior in visual quality and diversity compared to generation-by-optimization. In this work, we advance them in two significant ways. First, we introduce an instance normalization module to replace batch normalization with significant improvements to the quality of image stylization. Second, we improve diversity by introducing a new learning formulation that encourages generators to sample unbiasedly from the Julesz texture ensemble, which is the equivalence class of all images characterized by certain filter responses. Together, these two improvements take feed forward texture synthesis and image stylization much closer to the quality of generation-via-optimization, while retaining the speed advantage.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “We use instance normalization [45] across all layers of the network2.”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “Without data augmentation the model only achieves marginally better performance than the random network (which also achieves a non-negligible level of performance (Ulyanov et al., 2017; Caron et al., 2018)).”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “We run the code of improved texture networks [52] to perform single-style transfer, with IN or BN layers.”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “In Appendix C, we quantitatively show that Instance Normalization can indeed make the VGG features more domain-invariant.”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, methodology): “For example, LR is suitable for recurrent neural networks and language applications (Ba et al., 2016; Xu et al., 2019), while IN is widely used in image stylization and image generation (Ulyanov et al., 2016; 2017).”
Universal Style Transfer via Feature Transforms (2017, methodology): “Numerous methods have since been developed to address different aspects including speed [27, 19, 16], quality [28, 18, 32, 31], user control [10], diversity [29, 20], semantics understanding [7, 1] and photorealism [23].”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, methodology): “We apply Instance Normalization (IN) [47] to the content encoder E c , while we use Adaptive Instance Normalization (AdaIN) [19] and Layer Normalization (LN) [2] for the decoder G .”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, methodology): “and can lead to blurry results. Recently, the deep learning community has found that the VGG classication network [68] trained on ImageNet dataset [13] can be re-purposed as a \perceptual loss" [15,18,33,74,86,51], which can be used in paired image translation tasks [7,58,76], and was known to ”
Boundless: Generative Adversarial Networks for Image Extension (2019, background): “We additionally added instance normalization [39] after every generator layer besides the output layer, finding that it significantly reduces the number of artifacts in the generated images.”
Neural Style Transfer: A Review (2017, methodology): “Other than instance normalization, in [46], Ulyanov et al . also explore how to improve the diversity of the stylized images.”
LF-Net: Learning Local Features from Images (2018, methodology): “We convert them to grayscale for simplicity and simply normalize them individually using their mean and standard deviation [42].”
CHAOS Challenge - Combined (CT-MR) Healthy Abdominal Organ Segmentation (2020, positive): “(Shuo Han) • 3D network adopting U-Net variant in [36] and differs from sU-Net in [31], by: 1) A pre-activation residual block in each scale level at the encoder, 2) Convolutions with stride 2 to reduce the spatial size, 3) Instance normalization [37].”
EnlightenGAN: Deep Light Enhancement Without Paired Supervision (2019, methodology): “Furthermore, We add an instance normalization layer [39] after the VGG feature maps before feeding into L SFP and L LocalSFP in order to stabilize training.”
Toward Characteristic-Preserving Image-based Virtual Try-On Network (2018, methodology): “Each convolutional layer is followed by an Instance Normalization layer [33] and Leaky ReLU [24], of which the slope is set to 0.”
From BoW to CNN: Two Decades of Texture Representation for Texture Classification (2018, background): “The work of Gatys et al. [65] ignited a renewed interest in texture synthesis [229].”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, background): “IN has been extensively investigated in the field of image style transfer [177], [194], [195], where the style”