Learning Blind Video Temporal Consistency

By Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, Ming-Hsuan Yang. European Conference on Computer Vision, 2018

486 citations recorded by the source corpus.

Applying image processing algorithms independently to each frame of a video often leads to undesired inconsistent results over time. Developing temporally consistent video-based extensions, however, requires domain knowledge for individual tasks and is unable to generalize to other applications. In this paper, we present an efficient approach based on a deep recurrent network for enforcing temporal consistency in a video. Our method takes the original and per-frame processed videos as inputs to produce a temporally consistent video. Consequently, our approach is agnostic to specific image processing algorithms applied to the original video. We train the proposed network by minimizing both short-term and long-term temporal losses as well as a perceptual loss to strike a balance between temporal coherence and perceptual similarity with the processed frames. At test time, our model does not require computing optical flow and thus achieves real-time speed even for high-resolution videos. We show that our single model can handle multiple and unseen tasks, including but not limited to artistic style transfer, enhancement, colorization, image-to-image translation and intrinsic image decomposition. Extensive objective evaluation and subject study demonstrate that the proposed approach performs favorably against the state-of-the-art methods on various types of videos.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/098b68fe34dba7fc4e206035ae2d149944bbca8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Review of Single-Source Deep Unsupervised Visual Domain Adaptation (2020, unknown): “Further, maintaining the temporal consistency [216]”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, methodology): “In motion analysis, forward-backward consistency check can be used for detecting occlusion [28, 20, 52] or learning visual correspondence [45].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “We first consider the warping error, which is widely used in previous blind temporal consistency meth-ods [31,32,42].”
Softmax Splatting for Video Frame Interpolation (2020, background): “Temporal consistency is a common concern when synthesizing images in time [1, 16, 17, 26].”
Deep Video Inpainting (2019, methodology): “[14] proposed a deep CNN model that takes both original and per-frame processed videos as input and produces a temporally consistent video.”
Consistent video depth estimation (2020, methodology): “2018a], or application-agnostic post-processing algorithms [Bonneel et al. 2015; Lai et al. 2018].”
Channel Attention Is All You Need for Video Frame Interpolation (2020, background): “…and segmentation (Cheng et al. 2017; Zhu et al. 2018; 2017), video prediction (Li et al. 2018; Reda et al. 2018), and others (Chen et al. 2017; Lai et al. 2018)—frequently depend on the accurate optical ﬂow estimation since it leads to the success of the target tasks in terms of…”
Seeing Motion in the Dark (2019, methodology): “, SID [9] for per-frame processing, followed by learned blind temporal consistency [24]) therefore yields severe visible artifacts, as shown in Fig.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, background): “Consistent video stylization A common approach for video stylization involves applying image editing techniques (e.g., style transfer) on a frame-by-frame basis, followed by post-processing to address temporal inconsistencies in the resulting video [21, 25, 24].”
High Speed and High Dynamic Range Video with an Event Camera (2019, background): “Our temporal consistency loss is based on [35].”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “By processing frames individually using image style transfer [13,22], some works also learn to reduce the temporal consistency [5,27,29,30] in a post-process way.”