Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks

By Alex J. Champandard. 2016

251 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have proven highly effective at image synthesis and style transfer. For most users, however, using them as tools can be a challenging task due to their unpredictable behavior that goes against common intuitions. This paper introduces a novel concept to augment such generative architectures with semantic annotations, either by manually authoring pixel labels or using existing solutions for semantic segmentation. The result is a content-aware generative algorithm that offers meaningful control over the outcome. Thus, we increase the quality of images generated by avoiding common glitches, make the results look significantly more plausible, and extend the functional range of these algorithms---whether for portraits or landscapes, etc. Applications include semantic style transfer and turning doodles with few colors into masterful paintings!

Subfields: Cognitive Neuroscience, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Unrolled Generative Adversarial Networks (2016, methodology): “We optimize with 3 random starts using LBFGS, which is the optimizer typically used in similar settings such as style transfer (Johnson et al., 2016; Champandard, 2016).”
Neural Photo Editing with Introspective Adversarial Networks (2016, background): “Another related interface (Champanard, 2016) refines simple user input into complex textures through use of artistic style transfer (Gatys et al., 2015).”
Scribbler: Controlling Deep Image Synthesis with Sketch and Color (2016, background): “Examples of control signals include 3d pose of objects [8], natural language [39], semantic attributes [50], semantic segmentation [5], and object key-”
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis (2016, background): “Style Transfer In order to create realistic image textures, our work is motivated by the recent success of neural style transfer [15, 16, 28, 3, 39, 22].”
Universal Style Transfer via Feature Transforms (2017, methodology): “Numerous methods have since been developed to address different aspects including speed [27, 19, 16], quality [28, 18, 32, 31], user control [10], diversity [29, 20], semantics understanding [7, 1] and photorealism [23].”
Deep Photo Style Transfer (2017, methodology): “• We introduce an optional guidance to the style transfer process based on semantic segmentation of the inputs (similar to [2]) to avoid the content-mismatch problem, which greatly improves the photorealism of the results.”
I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence (2018, background): “It is also being applied in creative areas considered unique to humans, such as writing [4, 51], musical composition [7, 22], and drawing [5].”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “[1] discovered that the features extracted from a well-trained deep convolutional neural network can indicate the content structures, and their statistical distributions capture the style patterns, which inspired a line of works with advanced style and content descriptions [2, 3, 4, 5, 6, 7].”
Two-Stream Convolutional Networks for Dynamic Texture Synthesis (2017, background): “This has been explored previously for artistic style transfer [4, 14] with static imagery.”
Controlling Perceptual Factors in Neural Style Transfer (2016, methodology): “We use Johnson’s excellent publicly-available implementation of Fast Neural Style Transfer [13] 2 .”
Controllable Artistic Text Style Transfer via Shape-Matching GAN (2019, methodology): “Texture Transfer (GT ) Given the structure transfer result IX` = GS(I, `), the texture rendering task can be formulated as a standard image analogy problem such that X : Y :: IX` : I Y ` [14], which can be well solved by existing algorithms like the greedy-based Image Analogy [14] and the optimizati”
Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer (2018, methodology): “Most approaches utilize Gram matrices to capture the style of an image [23, 24, 87, 13], while some utilize an MRF framework to manipulate image patches in order to match the desired style [44, 10].”
Coherent Online Video Style Transfer (2017, methodology): “[18] on neural style transfer, there has been a number of recent works [38, 29, 9, 19] addressing the problem of style transfer using deep neural networks.”
Improving the Neural Algorithm of Artistic Style (2016, background): “An interesting application of [14] can be found in [4], where a rough sketch can be transformed into a painting using high-level annotations of an example image.”
Exploring the Neural Algorithm of Artistic Style (2016, background): “ while giving more photorealistic results (with a photorealistic style image as input), a good semantic correspondence between two images is needed. An interesting application of [14] can be found in [4], where a rough sketch can be transformed into a painting using high-level annotations of an exam”
Disentangled Representation Learning for Non-Parallel Text Style Transfer (2018, background): “In other work, researchers adopt data augmentation techniques to learn a disentangled latent space (Kulkarni et al. 2015; Champandard 2016).”