Fast Patch-based Style Transfer of Arbitrary Style

By T. Chen, Mark W. Schmidt. 2016

404 citations recorded by the source corpus.

Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “There are some recent works addressing this problem, but they are either still limited to a finite set of styles [11, 32, 55, 5], or much slower than the single-style transfer methods [6].”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “10, we compare out method with classical style transfer algorithms including Gatys et al. [5], Chen et al. [85], AdaIN [54], and WCT [55].”
Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, methodology): “Developed in the past, style transfer methods were mostly based on a texture transfer, where one copy the small fragments of one image to another [34].”
Generative Image Inpainting with Contextual Attention (2018, background): “Several works [35, 38] follow ideas from image stylization [5, 24] to formulate the inpainting as an optimization problem.”
Contextual-Based Image Inpainting: Infer, Match, and Translate (2017, methodology): “[5] proposes a fast approximation to [21] where it constructs the feature map directly and uses an inverse network to synthesize the image in feed-forward manner.”
Universal Style Transfer via Feature Transforms (2017, methodology): “[3] first propose to swap the content feature with the closest style feature locally.”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “[3] proposed a style-swap method, which is a patch based style transfer method relying on similarities between content and style patches.”
Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer (2018, background): “Others improve efficiency by avoiding the direct manipulation of the image and pre-training a model using large amounts of training data for a specific image style [35, 77, 45, 11, 90].”
BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models (2022, background): “Numerous problems in computer vision and graphics can be formulated as image-to-image translation problems, such as style transfer [3,9,13,22], semantic image synthesis [21,24,34,36,37, 40] and sketch-to-photo synthesis [2, 14, 43].”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, methodology): “Comprehensive evaluations have been conducted to compare with the prior style transfer methods [12, 4, 19, 13], our method achieves the state-of-the-art performance in terms of both visual quality and efficiency.”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, methodology): “[4] work on the overlapping patches extracted from the content image, swapping the features of the original patch with the features of the most similar patch in the style image, and then averages the features in the overlapping regions, thus producing an over-smoothed image without fine details (Fig”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “The existing attempts either tried to adjust the holistic statistics of the content features with that of the style features [17, 18, 19, 20, 21, 22, 23], or non-locally swap the relevant style features in order to match the content features [24, 25, 26, 27, 28, 29, 30, 31].”
The Contextual Loss for Image Transformation with Non-Aligned Data (2018, background): “One approach is to replace deep features of the input image with matching features of the target and then invert the features via efficient optimization [20] or through a pre-trained decoder [25].”
Shift-Net: Image Inpainting via Deep Feature Rearrangement (2018, methodology): “In [4], local matching is performed on the convolution layer of the pre-trained network to combine content and style, and an inverse network is then deployed to generate the image from feature representation.”
Multi-style Generative Network for Real-time Transfer (2017, background): “A style swap layer is proposed in [5], but gets lower quality and slower speed (compared to existing feed-forward approaches).”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modiﬁes color distributions and texture patterns of content images.”
Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation (2020, methodology): “We first diversity the texture of the original source dataset with a style transfer algorithm Style-swap [4] and translate the original source dataset with an image translation algorithm CycleGAN [29].”