Detail-Revealing Deep Video Super-Resolution

By Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, Jiaya Jia. IEEE International Conference on Computer Vision, 2017

564 citations recorded by the source corpus.

Previous CNN-based video super-resolution approaches need to align multiple frames to the reference. In this paper, we show that proper frame alignment and motion compensation is crucial for achieving high quality results. We accordingly propose a “sub-pixel motion compensation” (SPMC) layer in a CNN framework. Analysis and experiments show the suitability of this layer in video SR. The final end-to-end, scalable CNN framework effectively incorporates the SPMC layer and fuses multiple frames to reveal image details. Our implementation can generate visually and quantitatively high-quality results, superior to current state-of-the-arts, without the need of parameter tuning.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2b5c2314b79e19205e4f0f91f7c8d8b12c337d7f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “We compare our EDVR method with nine algorithms: RCAN [51], DeepSR [19], BayesSR [21], VESPCN [2], SPMC [37], TOFlow [48], FRVSR [32], DUF [10] and RBPN [6] on three testing datasets: Vid4 [21], Vimeo-90K-T [48] and REDS4.”
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, methodology): “Bicubic SPMC [27] FSRVSR [22] Model Sizes: Table 4 shows parameter numbers of several networks with the leading VSR performance.”
Crowd Counting and Density Estimation by Trellis Encoder-Decoder Networks (2019, methodology): “In hourglass encoder-decoder networks with a single decoding path [33, 54, 6, 42], features must endure excessive down-sampling and up-sampling operations, which degrade the pixel-wise precision.”
VRT: A Video Restoration Transformer (2022, methodology): “Early methods [5, 35, 51, 54, 88] use traditional flow estimation methods to estimate optical flow and warp the supporting frames towards the reference frame.”
Scale-Recurrent Network for Deep Image Deblurring (2018, methodology): “2(a)), also termed as encoder-decoder networks [24], which greatly increases regression ability and is widely used in recent work of FlowNet [10], video deblurring [33], video super-resolution [35], frame synthesis [23], etc.”
Video Enhancement with Task-Oriented Flow (2017, background): “ng, researchers have directly modeled video enhancement tasks as regression problems without representing motions, and have designed deep networks for frame interpolation [21,24] and super-resolution [14,10,12,29]. Recently, with differentiable image sampling layers in deep learning [11], motion inf”
Efficient Long-Range Attention Network for Image Super-resolution (2022, background): “In recent years, deep convolutional neural network (CNN) [24] based SR models [11, 12, 48, 30, 6, 51, 65, 47, 55] have become prevalent for their strong capability in recovering or generating [25, 56] image high-frequency details, showing high practical value in image and video restoration, transiti”
Burst Denoising with Kernel Prediction Networks (2017, background): “The success of deep learning has yielded a number of neural network approaches to multi-image denoising [29, 27], in addition to a wide range of similar tasks such as joint denoising and demosaicking [7], deblurring [24], and superresolution [25].”
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, background): “Video super-resolution, by nature, involves a long and complex processing pipeline since it needs to aggregate information from not only the spatial dimension but also the temporal dimension.”
Infrared and Visible Image Fusion with ResNet and zero-phase component analysis (2018, unknown): “Furthermore, in image reconstruction tasks [23] [24] [25], the performance gets better by apply the residual block.”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “…examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical ﬂow prediction [32, 47, 52], optical ﬂow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement [7, 46, 51].”
Wide-Context Semantic Image Extrapolation (2019, background): “Current SR frameworks either upsample input before fed into networks [8], or use spatial expansion modules [24, 36, 38] within the network.”
Frame-Recurrent Video Super-Resolution (2018, methodology): “The latest state-of-the-art video super-resolution methods approach the problem by combining a batch of LR frames to estimate a single HR frame, effectively dividing the task of video super-resolution into a large number of separate multi-frame super-resolution subtasks [3, 26, 27, 36].”
Deep Learning for Image Super-Resolution: A Survey (2019, methodology): “[196] root from accurate LR imaging model and propose a sub-pixel-like module to simultaneously achieve motion compensation and super-resolution, and thus fuse the aligned frames more effectively.”
Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation (2018, methodology): “[34] used motion compensation transformer module from VESPCN [1] for the motion estimation, and proposed a subpixel motion compensation layer for simultaneous motion compensation and upsampling.”
Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (2019, methodology): “With recent development of deep learning, a lot of video SR methods [3, 14, 23, 26, 31] based on convolutional neural networks (CNNs) have emerged.”
Recurrent Back-Projection Network for Video Super-Resolution (2019, methodology): “In RNNs [30, 27, 13], it is not easy to jointly model subtle and significant changes (e.”