Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation

By Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, Seon Joo Kim. 2018

600 citations recorded by the source corpus.

Video super-resolution (VSR) has become even more important recently to provide high resolution (HR) contents for ultra high definition displays. While many deep learning based VSR methods have been proposed, most of them rely heavily on the accuracy of motion estimation and compensation. We introduce a fundamentally different framework for VSR in this paper. We propose a novel end-to-end deep neural network that generates dynamic upsampling filters and a residual image, which are computed depending on the local spatio-temporal neighborhood of each pixel to avoid explicit motion compensation. With our approach, an HR image is reconstructed directly from the input image using the dynamic upsampling filters, and the fine details are added through the computed residual. Our network with the help of a new data augmentation technique can generate much sharper HR videos with temporal consistency, compared with the previous methods. We also provide analysis of our network through extensive experiments to show how the network deals with motions implicitly.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Sa & Sp & Te Classification [196], [197], Frame interpolation [234], [235], Super resolution [236], Video deblurring [237], [238], Action prediction [239]”
A Deep Journey into Super-resolution (2019, background): “For SR in higher dimensional inputs (such as videos and 3D scans), we refer the reader to recent seminal works [37, 51, 79, 83, 88].”
NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study (2019, methodology): “They trained an ensemble model that generates a spatial weight map for RDN [42], RCAN [41], and DUF [9].”
Meta-SR: A Magnification-Arbitrary Network for Super-Resolution (2019, background): “[12] introduced the dynamic upsampling filters for video super-resolution.”
VRT: A Video Restoration Transformer (2022, methodology): “Jo et al. [34] use dynamic upsampling filters for implicit motion compensation.”
Recurrent Back-Projection Network for Video Super-Resolution (2019, methodology): “(a) Input frames are concatenated to preserve temporal information [18, 2, 16, 22].”
Deep Learning for Image Super-Resolution: A Survey (2019, background): “And Jo et al. [196] generate dynamic upsampling ﬁlters and the HR residual image based on the local spatio-temporal neighborhood of each pixel, and also avoid explicit motion compensation.”
Spatio-Temporal Filter Adaptive Network for Video Deblurring (2019, background): “[13] reconstructs high-resolution image from low-resolution input using generated dynamic upsam-pling ﬁlters.”
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, methodology): “DUF [16] leverages dynamic upsampling filters to handle motions implicitly.”
EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “We compare our EDVR method with nine algorithms: RCAN [51], DeepSR [19], BayesSR [21], VESPCN [2], SPMC [37], TOFlow [48], FRVSR [32], DUF [10] and RBPN [6] on three testing datasets: Vid4 [21], Vimeo-90K-T [48] and REDS4.”
Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis (2019, methodology): “This idea has been applied to different applications such as neural style transfer [27], superresolution [15, 12], image segmentation [8, 30], motion prediction [32] and tracking [19].”
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, methodology): “In comparison with SPMC [27] and DUF [9], beneﬁting from the strong temporal alignment network, the proposed TDAN is more effective in exploiting information in supporting frames, and therefore it is more capable of restoring image structures, for example, the baby face and the stripe on the cloth…”
Context-Gated Convolution (2019, methodology): “The family of work on dynamic filters [28,29,35] generates weights of convolution kernels using features extracted from input images by another CNN feature extractor.”
Video Enhancement with Task-Oriented Flow (2017, background): “2017; Niklaus and Liu 2018), super-resolution (Huang et al. 2015; Kappeler et al. 2016; Tao et al. 2017; Bulat et al. 2018; Ahn et al. 2018; Jo et al. 2018), denoising Mildenhall et al.”
Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (2019, methodology): “Space-based methods [3, 13, 14] take multiple frames as supplementary materials to help reconstruct the reference frame.”
Hyper-LifelongGAN: Scalable Lifelong Learning for Image Conditioned Generation (2021, methodology): “This idea has been applied to applications in different research fields such as few shot learning [27, 33], image segmentation [2, 36] and generative models [28, 18, 16], which is the focus of our paper.”
CARAFE: Content-Aware ReAssembly of FEatures (2019, background): “Within the realms of super-resolution and denoising, some other works [27, 16, 11] also explore using learnable kernels spatially in low-level vision.”