EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

By Xintao Wang, Kelvin C. K. Chan, K. Yu, Chao Dong, Chen Change Loy. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019

1,302 citations recorded by the source corpus.

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/28b3833743ab00904da1f4a30cd6c771cc164c0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study (2019, positive): “They proposed the EDVR framework [33].”
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, methodology): “The profile from EDVR possesses noise, indicating flickering artifacts.”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “[9] proposed to use deformable convolution together with the learnt offset maps to enhance the modeling capability of convolution neural networks, which has achieved promising results in several videorelated tasks like action recognition [18] and video superresolution [30, 35].”
FaPN: Feature-aligned Pyramid Network for Dense Image Prediction (2021, neutral): “To establish accurate correspondences among multiple frames given a large motion for video restoration, TDAN [42] and EDVR [43] achieve implicit motion compensation with by deformable convolution [10] at the feature level.”
Artificial intelligence in the creative industries: a review (2020, methodology): “ompensation. The original target frame is fed, along with its neighbouring frames, into intermediate layers of the CNN to perform inter-frame motion compensation during feature extraction [103]. EDVR [100], the winner of the NTIRE19 video restoration and enhancement challenges in 201950, employs a d”
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, background): “After that, it has already made a good impact on our community and been followed and improved by recent works including EDVR [11] for video super-resolution and deblurring and Zooming SlowMo [12] for space-time video super-resolution.”
Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (2019, methodology): “However, these methods [6, 12, 35] are too complex, for instance, Wang et al.”
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, methodology): “9458 EDVR-M [32] 3.”
AIM 2020 Challenge on Real Image Super-Resolution: Methods and Results (2020, methodology): “ss function proposed by Jon Barron was used. For track 3, it additionally used a spatial attention module and an e- cient channel attention module. The spatial attention module is borrowed from EDVR [30] and the ecient attention module is borrowed from ECA-Net [29]. Considering that the training dat”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “Other works [53, 85, 91, 97] focus on restoring the quality of low bit-rate videos using deep networks.”
Deep Image Deblurring: A Survey (2022, methodology): “Numerousdeepvideodeblurringalgorithmsbasedon this architecture have been developed (Nah et al. 2019; Sim and Kim 2019; Wang et al. 2019).”
Efficient Long-Range Attention Network for Image Super-resolution (2022, background): “In recent years, deep convolutional neural network (CNN) [23] based SR models [11, 12, 47, 29, 6, 50, 60, 46, 53] have become prevalent for their strong capability in recovering or generating [24, 54] image high-frequency details, showing high practical value in image and video restoration,…”
VRT: A Video Restoration Transformer (2022, methodology): “2) Temporal Alignment and Fusion: Since supporting frames are often highly-related but misaligned, temporal alignment plays an critical role in video restoration [2], [3], [12], [13], [31].”
Changer: Feature Interaction is What You Need for Change Detection (2022, methodology): “Moreover, alignment-based fusion methods focus on feature alignment, and they typically use ﬂow ﬁeld or de-formable convolution [10, 50] to align features of different levels in the spatial dimension [16,17,21,34,38].”
Light Field Super-Resolution with Zero-Shot Learning (2021, background): “, attention [35] and deformable offsets [37], while the latter cast the redundancy as geometry correspondences, e.”
A Self-supervised Approach for Adversarial Robustness (2020, background): “Generator (Pθ): Our generator architecture is inspired by [24, 44].”
Neural Video Compression with Diverse Contexts (2023, background): “The works [9,54,58] show deformable alignment gets better results for video restoration as each location has multiple offsets to capture temporal correspondence.”
A Deep Journey into Super-resolution (2019, background): “resolution (SISR) which has been extensively studied due to its challenging nature. For SR in higher dimensional inputs (such as videos and 3D scans), we refer the reader to recent seminal works [1], [2], [3], [4], [5]. High-resolution images provide improved reconstructed details of the scenes and ”