StyleBank: An Explicit Representation for Neural Image Style Transfer

By Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, Gang Hua. 2017

499 citations recorded by the source corpus.

We propose StyleBank, which is composed of multiple convolution filter banks and each filter bank explicitly represents one style, for neural image style transfer. To transfer an image to a specific style, the corresponding filter bank is operated on top of the intermediate feature embedding produced by a single auto-encoder. The StyleBank and the auto-encoder are jointly learnt, where the learning is conducted in such a way that the auto-encoder does not encode any style information thanks to the flexibility introduced by the explicit filter bank representation. It also enables us to conduct incremental learning to add a new image style by learning a new filter bank while holding the auto-encoder fixed. The explicit style representation along with the flexible network design enables us to fuse styles at not only the image level, but also the region level. Our method is the first style transfer network that links back to traditional texton mapping methods, and hence provides new understanding on neural style transfer. Our method is easy to train, runs in real-time, and produces results that qualitatively better or at least comparable to existing methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

GAN Memory with No Forgetting (2020, background): “Style-transfer techniques An extensive literature [23, 71, 79, 13, 62, 38] has explored how one can manipulate the style of an image (e.”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Moreover, images can be stylized by disentangling the style and the content [13, 21] or by encoding styles into a stylebank (set of convolution filters) [7].”
Universal Style Transfer via Feature Transforms (2017, background): “Since then, significant efforts have been made to synthesize stylized images by minimizing Gram/covariance matrices based loss functions, through either iterative optimization [9] or trained feed-forward networks [27, 16, 20, 2, 6].”
The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution (2018, background): “Building on fast style transfer, [36, 37] proposed models to transfer textures from multiple style images.”
Dynamic Multi-Scale Filters for Semantic Segmentation (2019, background): “[4] explicitly constructs filters bank encoded with objective content which can transform the input image to a specific style.”
A Closed-form Solution to Photorealistic Image Stylization (2018, methodology): “The major step in the algorithm is to solve an optimization problem of matching the Gram matrices of deep features extracted from the content and style photos. A number of methods have been developed [15,16,17,18,19,20,21,22,10,23] to further improve its stylization performance and speed. However, t”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “Recently, a number of works [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] approximated the iterative back-propagation procedure as the feed-forward networks for real-time style transfer.”
Dynamic Instance Normalization for Arbitrary Style Transfer (2019, methodology): “To address this issue, Multiple-StylePer-Model (MSPM) algorithms are proposed by incorporating multiple styles into one single model (Zhang and Dana 2017; Chen et al. 2017; Li et al. 2017b; Dumoulin, Shlens, and Kudlur 2017).”
Controllable Artistic Text Style Transfer via Shape-Matching GAN (2019, methodology): “trained a feed-forward StyleNet [18] using the loss of Neural Style Transfer [10] for fast style transfer [27, 15, 22, 21, 6].”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, methodology): “Such deep features have been widely used in image generation [5,9,24,26,27,36,39,63] with the goal of synthesizing indistinguishable realistic images.”
Deep Learning for Text Style Transfer: A Survey (2020, background): “Note that there are style transfer works across different modalities, including images (Gatys, Ecker, and Bethge 2016; Zhu et al. 2017; Chen et al. 2017b), text, voice (Gao, Singh, and Raj 2018; Qian et al. 2019; Yuan et al. 2021), handwriting (Azadi et al. 2018; Zhang and Liu 2013), and videos…”
Rethinking and Improving the Robustness of Image Style Transfer (2021, methodology): “A consistent observation from this work [43, 24, 30, 23, 25, 19, 9, 2, 8] is that the correlation between the activations of a pre-trained VGG [33] network has remarkable ability to capture the visual style of an image.”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, methodology): “[8, 9] discovered that multilevel feature statistics extracted from a trained CNN notably represent the characteristics of visual styles, which boosts the development of style transfer approaches, either by iterative optimizations [9, 16, 30, 20] or feed-forward networks [5, 13, 18, 20, 31, 28, 29, ”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, background): “Beyond that, a number of methods have been proposed to address different aspects of style transfer, including quality [14,49,4,47,23], diversity [28,44], photorealism [32], combining several styles in a single model [45,7,3] and generalizing to previously unseen styles [21,29,15,38].”
Neural Style Transfer via Meta Networks (2018, background): “[2] built a explicit representation for each style.”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “In the literature, there are Multiple-Style-Per-Model solutions [7, 2, 23, 44] and Arbitrary-Style-Per-Model [14, 3, 22, 28, 15, 24, 6, 5, 43, 32, 42, 11] methods.”
Coherent Online Video Style Transfer (2017, methodology): “, [25, 10]), our method is able to produce consistent and stable stylized video sequences in near real-time.”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modiﬁes color distributions and texture patterns of content images.”