Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses

By Pierre Wilmot, Eric Risser, Connelly Barnes. 2017

261 citations recorded by the source corpus.

Recently, methods have been proposed that perform texture synthesis and style transfer by using convolutional neural networks (e.g. Gatys et al. [2015,2016]). These methods are exciting because they can in some cases create results with state-of-the-art quality. However, in this paper, we show these methods also have limitations in texture quality, stability, requisite parameter tuning, and lack of user controls. This paper presents a multiscale synthesis pipeline based on convolutional neural networks that ameliorates these issues. We first give a mathematical explanation of the source of instabilities in many previous approaches. We then improve these instabilities by using histogram losses to synthesize textures that better statistically match the exemplar. We also show how to integrate localized style losses in our multiscale framework. These losses can improve the quality of large features, improve the separation of content and style, and offer artistic controls such as paint by numbers. We demonstrate that our approach offers improved quality, convergence in fewer iterations, and more stability over the optimization.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “The gram matrix is adopted on the level of feature map to encode such semantically-irrelevant information following prior works [8, 13, 23, 24, 31].”
Style Transfer by Relaxed Optimal Transport and Self-Similarity (2019, methodology): “Perhaps the most common form of innovation is in proposals for quantifying the ’stylistic similarity’ between two images [15, 2, 22, 20].”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, background): “Beyond that, a number of methods have been proposed to address different aspects of style transfer, including quality [14,49,4,47,23], diversity [28,44], photorealism [32], combining several styles in a single model [45,7,3] and generalizing to previously unseen styles [21,29,15,38].”
Controlling Perceptual Factors in Neural Style Transfer (2016, background): “Note that concurrent work [27] independently developed a similar approach for spatial control as presented here.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “Other effective loss functions have been proposed, such as MRF loss [30], adversarial loss [31], histogram loss [54], CORAL loss [41], MMD loss [33], and distance between channel-wise mean and variance [33].”
The Contextual Loss for Image Transformation with Non-Aligned Data (2018, background): “In [28] a histogram loss was suggested in order to synthesize textures that match the target statistically.”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “Despite visually plausible results are reported in these works, the careful hyper-parameter tuning process is often required [39, 40] to better optimize the objective function for each style.”
Arbitrary Style Transfer With Style-Attentional Networks (2018, methodology): “Despite recent advances, existing methods [5, 4, 1, 8, 12, 22, 3, 6, 10, 11, 23, 24, 28, 18] present a trade-off among generalization, quality, and efficiency.”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, methodology): “Since the optimization-based approaches render visually plausible results for various styles at the sacrifice of executing efficiency[9, 16, 30], while the feed-forward networks are either restricted by a finite set of predefined styles [5, 13, 18, 20, 31, 28], or simplify the zero-shot transfer wit”
StyTr2: Image Style Transfer with Transformers (2021, methodology): “Optimizationbased style transfer methods [19, 31, 47] render the input content images with learned style representation iteratively.”
Language-Driven Image Style Transfer (2021, background): “Artistic Style Transfer Style transfer [1, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56] is to redraw an image with a speciﬁc style.”
Neural Style Transfer via Meta Networks (2018, background): “There are also other representations of the style, such as histogram loss [30], MRF loss [20] and CORAL loss [29].”
Neural Style Transfer: A Review (2017, background): “However, feature activations with quite different means and variances can still have the same Gram matrix (Figure 4 in [38]).”