Coherent Online Video Style Transfer

By Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, Gang Hua. IEEE International Conference on Computer Vision, 2017

311 citations recorded by the source corpus.

Training a feed-forward network for the fast neural style transfer of images has proven successful, but the naive extension of processing videos frame by frame is prone to producing flickering results. We propose the first end-toend network for online video style transfer, which generates temporally coherent stylized video sequences in near realtime. Two key ideas include an efficient network by incorporating short-term coherence, and propagating short-term coherence to long-term, which ensures consistency over a longer period of time. Our network can incorporate different image stylization networks and clearly outperforms the per-frame baseline both qualitatively and quantitatively. Moreover, it can achieve visually comparable coherence to optimization-based video style transfer, but is three orders of magnitude faster.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2d5b4144124f47a6357e6390dc6c0f8806ac54f5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Blind Video Temporal Consistency (2018, background): “[7] learn flow and mask networks to adaptively blend the intermediate features of the pre-trained style network.”
Video-to-Video Synthesis (2018, background): “Video style transfer [10,22,28,58], transferring the style of a reference painting to a natural scene video, is also related.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, background): “…2018), object detection and segmentation (Cheng et al. 2017; Zhu et al. 2018; 2017), video prediction (Li et al. 2018; Reda et al. 2018), and others (Chen et al. 2017; Lai et al. 2018)—frequently depend on the accurate optical ﬂow estimation since it leads to the success of the target tasks in…”
Consistent video depth estimation (2020, background): “In light of this, various approaches for enforcing temporal consistency have been developed in the context of style transfer [Chen et al. 2017; Huang et al. 2017; Ruder et al. 2016], image-based graphics applications [Lang et al.”
Frame-Recurrent Video Super-Resolution (2018, methodology): “[4] pass on a feature representation to the next step, Gupta et al .”
Deep Learning for Text Style Transfer: A Survey (2020, background): “Note that there are style transfer works across different modalities, including images (Gatys, Ecker, and Bethge 2016; Zhu et al. 2017; Chen et al. 2017b), text, voice (Gao, Singh, and Raj 2018; Qian et al. 2019; Yuan et al. 2021), handwriting (Azadi et al. 2018; Zhang and Liu 2013), and videos…”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, background): “roaches render visually plausible results for various styles at the sacriﬁce of executing efﬁciency[11,18,32], while the feed-forward networks are either restricted by a ﬁnite set of predeﬁned styles [7,15,20,22,33,30,3,5], or simplify the zero-shot transfer with compromised visual quality [14,27,31”
Gated Context Aggregation Network for Image Dehazing and Deraining (2018, background): “In the future, we will try more facy losses used in [6, 19] and consider to extend to video dehazing like [5].”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, unknown): “Still, the efficient test-time inference has led to their widespread adoption as a sub-module in applications requiring to process tempo ral information, including video object segmentation [13], video recognition [15, 43, 64], and video style transfer [11].”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “This can be alleviated by adding temporal coherency loss [Chen et al. 2017] when training our disentangled attribute-latent mapping network.”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “Video style transfer takes a reference style image and statistically applies its style to an input video [40, 8, 55].”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “, [30] for optimization based video style transfer, [31, 1, 12, 13, 8] for per-style-per-model methods, [37, 38] for arbitrary-style-per-model methods, and [36, 4, 26] for image-to-image translation frameworks.”
A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (2017, background): “Inspired by the tremendous success of deep learning for large-scale visual recognition tasks like ILSVRC [31, 23], a variety of recent work has investigated deep neural networks for low-level computer vision tasks such as image denoising [30, 14], shadow removal [18], image smoothing [41, 27] and st”
Deep Video Inpainting (2019, methodology): “The mask sub-network consists of three convolution layers and takes the absolute difference of the two feature maps |Fs′ − Fr| as input and produces single channel composition mask m, as suggested in [6].”
Neural Style Transfer: A Review (2017, background): “nsistency loss. Another concurrent work which shares a similar idea with [75] but with an additional exploration of style instability problem can be found in [76]. Different from [75,76], Chen et al. [77] propose a ﬂow subnetwork to produce feature ﬂow and incorporate optical ﬂow information in feat”
Artistic Style Transfer for Videos and Spherical Images (2017, background): “[2] warp and combine high-level feature maps from previous frames to improve consistency.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “Ewarp The warping error 𝐸 𝑤𝑎𝑟𝑝 is used by [4, 25, 32] to measure the temporal inconsistency for video style transfer.”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modifies color distributions and texture patterns of content images.”
Neural Style Transfer via Meta Networks (2018, background): “• Sample minibatch of m input images {I [1] c , .”