Multi-style Generative Network for Real-time Transfer

By Hang Zhang, Kristin J. Dana. 2017

306 citations recorded by the source corpus.

Despite the rapid progress in style transfer, existing approaches using feed-forward generative network for multi-style or arbitrary-style transfer are usually compromised of image quality and model flexibility. We find it is fundamentally difficult to achieve comprehensive style modeling using 1-dimensional style embedding. Motivated by this, we introduce CoMatch Layer that learns to match the second order feature statistics with the target styles. With the CoMatch Layer, we build a Multi-style Generative Network (MSG-Net), which achieves real-time performance. In addition, we employ an specific strategy of upsampled convolution which avoids checkerboard artifacts caused by fractionally-strided convolution. Our method has achieved superior image quality comparing to state-of-the-art approaches. The proposed MSG-Net as a general approach for real-time style transfer is compatible with most existing techniques including content-style interpolation, color-preserving, spatial control and brush stroke size control. MSG-Net is the first to achieve real-time brush-size control in a purely feed-forward manner for style transfer. Our implementations and pre-trained models for Torch, PyTorch and MXNet frameworks will be publicly available (Links can be found at http://hangzhang.org/).

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “It also drives research progress in multimodal learning and inference across vision and language, which is one of the most active research areas in recent years [20, 18, 36, 19, 41, 4, 30, 5, 1, 31, 33, 32]”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “There are some recent works addressing this problem, but they are either still limited to a finite set of styles [11, 32, 55, 5], or much slower than the single-style transfer methods [6].”
Context Encoding for Semantic Segmentation (2018, methodology): “This feedback strategy is inspired by prior work in style transfer [22, 57] and a recent work SE-Net [20] that tune featuremap scale or statistics.”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, methodology): “There are substantial existing methods investigating the better usage of GANs for different applications, such as image synthesis [33, 38], (unpaired) pixel-to-pixel translation [15, 52], medical applications [5, 50], etc [22, 45, 13, 46].”
Artistic Style Transfer for Videos and Spherical Images (2017, background): “s. Common to all approaches is the limitation that each model can only learn to transform the style according to one specic style template. However, recent techniques also allow multi-style networks [29,6]. Painted animations. Litwinowicz [16] proposed an approach to create temporally consistent art”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “Recently, a number of works [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] approximated the iterative back-propagation procedure as the feed-forward networks for real-time style transfer.”
Density-Aware Single Image De-raining Using a Multi-stream Dense Network (2018, background): “Recently, due to the immense success of deep learning in both high-level and low-level vision tasks [10, 34, 44, 23, 35, 37, 25], several CNN-based methods have also been proposed for image de-raining [4, 6, 36, 7].”
Neural Style Transfer: A Review (2017, methodology): “[48], 2 hours for the algorithm of Li and Wand [52], 4 hours for Zhang and Dana [56], and 8 hours for Li et al.”
Spatially Controllable Image Synthesis with Internal Representation Collaging (2018, methodology): “GANs has been particularly successful on the tasks on images: applications include image colorization [14, 16], inpainting [32, 15, 41], domain translation [16, 40, 46, 37, 39], style transfer [12, 42], object transﬁguration [45, 24, 13, 22], just to name a few.”
Dynamic Instance Normalization for Arbitrary Style Transfer (2019, methodology): “To address this issue, Multiple-StylePer-Model (MSPM) algorithms are proposed by incorporating multiple styles into one single model (Zhang and Dana 2017; Chen et al. 2017; Li et al. 2017b; Dumoulin, Shlens, and Kudlur 2017).”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “In the literature, there are Multiple-Style-Per-Model solutions [7, 2, 23, 44] and Arbitrary-Style-Per-Model [14, 3, 22, 28, 15, 24, 6, 5, 43, 32, 42, 11] methods.”
Neural Style Transfer via Meta Networks (2018, background): “Several other works on image transformation networks have been proposed ever since the neural art transfer has emerged, but these pre-trained image transformation networks are limited to single [33, 21] or several styles [5, 22, 35].”
A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation (2017, background): “Recent work on style-transfer [108], image de-raining [109] and image-to-image translation [42] have demonstrated promising re-sults from the use of additional loss functions such as adversarial loss and perceptual loss.”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, background): “This network matches image styles at multiple scale and puts the computational burden into training [210].”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, methodology): “A common practice is employing external style signals to supervise the content modification [12, 25, 31, 29, 3] on a feed-forward network.”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modiﬁes color distributions and texture patterns of content images.”
Generating High-Quality Crowd Density Maps Using Contextual Pyramid CNNs (2017, background): “It has been widely acknowledged that minimization of L2 error results in blurred results especially for image reconstruction tasks [13, 14, 45, 46, 47].”
Domain-Adaptive Few-Shot Learning (2020, methodology): “Further, we utilize the style transfer algorithm [63] to transfer the samples from Cd and Ct into a new domain.”