Video Enhancement with Task-Oriented Flow

By Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, William T. Freeman. International Journal of Computer Vision, 2017

1,736 citations recorded by the source corpus.

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c1045435c208a20f65b79baaa2d79783c2409c09

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Scale-Space Flow for End-to-End Optimized Video Compression (2020, neutral): “available in Vimeo-90K [39].”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “Video interpolation was the subject of a flurry of interest in the deep learning community a number of years ago [Niklaus et al., 2017, Jiang et al., 2018, Xue et al., 2019, Bao et al., 2019].”
CompressAI: a PyTorch library and evaluation platform for end-to-end compression research (2020, methodology): “Training Unless specified otherwise, the provided pre-trained networks were trained for 4-5M steps on 256× 256 image patches randomly extracted and cropped from the Vimeo-90K dataset [22].”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “We use the Vimeo-90K dataset [40] in the training stage, which contains 89,800 video clips with each video having 7 frames with the resolution of 448 × 256 .”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “There are 3,782 triplets in the Vimeo90K testing set [62] with resolution of 448 × 256.”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “Research on frame interpolation has seen a recent resurgence, with multiple papers proposing kernel-based [3, 4, 38, 39], ﬂow-based [3, 4, 22, 28, 29, 37, 41, 43, 51], and phase-based [34, 35] approaches.”
Recurrent Back-Projection Network for Video Super-Resolution (2019, methodology): “, 2× and 8×) on Vimeo90k [34], SPMCS-32 [30], and Vid4 [24] as shown in Table 10.”
EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “Most existing approaches perform alignment by explicitly estimating optical flow field between the reference and its neighboring frames [2, 48, 13].”
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, methodology): “[32] for releasing a VSR dataset: Vimeo Super-Resolution dataset containing 64612 training samples.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “…neural networks (CNNs) have enabled this process to be trained in an end-to-end manner, and many video frame interpolation techniques incorporate a CNN-based optical ﬂow module into their core networks (Bao et al. 2018; Jiang et al. 2018; Liu et al. 2017; Niklaus and Liu 2018; Xue et al. 2018).”
VRT: A Video Restoration Transformer (2022, methodology): “2) Temporal Alignment and Fusion: Since supporting frames are often highly-related but misaligned, temporal alignment plays an critical role in video restoration [2], [3], [12], [13], [31].”
DVC: An End-To-End Deep Video Compression Framework (2018, methodology): “We train the proposed video compression framework using the Vimeo-90k dataset [42], which is recently built for evaluating different video processing tasks,”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, background): “In fact, a recent work [30] demonstrated that fixed flow estimation is not as good as task-oriented flow for general computer vision tasks.”
Deep Contextual Video Compression (2021, methodology): “1 Experimental settings Training data We use the training part in Vimeo-90k septuplet dataset [33] as our training data.”
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (2021, background): “While some works solely focus on transferring a predefined holistic motion between objects [73] or interpolating motion between a starting and end frame [54, 79, 43, 4, 55], the most commonly addressed problem is video prediction.”