Style Transfer by Relaxed Optimal Transport and Self-Similarity

By Nicholas I. Kolkin, Jason Salavon, Gregory Shakhnarovich. Computer Vision and Pattern Recognition, 2019

326 citations recorded by the source corpus.

The goal of style transfer algorithms is to render the content of one image using the style of another. We propose Style Transfer by Relaxed Optimal Transport and Self-Similarity (STROTSS), a new optimization-based style transfer algorithm. We extend our method to allow user specified point-to-point or region-to-region control over visual similarity between the style image and the output. Such guidance can be used to either achieve a particular visual effect or correct errors made by unconstrained style transfer. In order to quantitatively compare our method to prior work, we conduct a large-scale user study designed to assess the style-content tradeoff across settings in style transfer algorithms. Our results indicate that for any desired level of content preservation, our method provides higher quality stylization than prior work.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6def889af5f88aa2d287625bf3ecd698f7a6133d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking and Improving the Robustness of Image Style Transfer (2021, background): “A consistent observation from this work [43, 24, 30, 23, 25, 19, 9, 2, 8] is that the correlation between the activations of a pre-trained VGG [33] network has remarkable ability to capture the visual style of an image.”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, background): “Many style transfer works fall under this category where the content of the input image must be preserved while the style of the reference image is transferred [13,29,16,14,26,20,31,30,38,1], yet they struggle to generate diverse images.”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modiﬁes color distributions and texture patterns of content images.”
Text2Mesh: Text-Driven Neural Stylization for Meshes (2021, background): “Editing visual data to conform to a desired style, while preserving the underlying content, is a longstanding objective in computer graphics and vision [14,21,23,24,30].”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, methodology): “ral photographs. The training only requires a single image from each domain. We compare our results (SinCUT) to recent style and photo transfer methods including Gatys et al. [18], WCT2 [81], STROTSS [38], and patch-based CycleGAN [88]. Our method generates can reproduce the texture of the reference”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Users were asked to choose between the left result and the right one, akin to the standard practice of Two-Alternative Forced Choice (2AFC) [8, 35, 42].”
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (2022, methodology): “These affinities have a tight connection to the established concept of self-similarly, which has been used to design structure descriptors by both classical and modern works [3,24,41,46].”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “We adopt the Two-alternative Forced Choice (2AFC) protocol, as used in previous works (Kolkin et al., 2019; Zhang et al., 2018; Blattmann et al., 2023a; Rombach et al., 2022).”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “STROTSS [45] is an optimization-based framework, in the spirit of the classic method of Gatys et al. [16].”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “We adopt the Two-alternative Forced Choice (2AFC) protocol suggested in [20,31].”
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis (2021, methodology): “The baseline is formed by using CLIPDraw [1] to convert the input text into an image, then performing style transfer [2].”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, methodology): “Thereafter a number of variants [3, 5, 37, 38, 7, 39, 40] have been developed based on new style and content losses, or by adopting this framework to different scenarios and requirements.”