TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution

By Yapeng Tian, Yulun Zhang, Yun Raymond Fu, Chenliang Xu. Computer Vision and Pattern Recognition, 2018

642 citations recorded by the source corpus.

Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). Due to varying motion of cameras or objects, the reference frame and each support frame are not aligned. Therefore, temporal alignment is a challenging yet important problem for VSR. Previous VSR methods usually utilize optical flow between the reference frame and each supporting frame to warp the supporting frame for temporal alignment. However, both inaccurate flow and the image-level warping strategy will lead to artifacts in the warped supporting frames. To overcome the limitation, we propose a temporally-deformable alignment network (TDAN) to adaptively align the reference frame and each supporting frame at the feature level without computing optical flow. The TDAN uses features from both the reference frame and each supporting frame to dynamically predict offsets of sampling convolution kernels. By using the corresponding kernels, TDAN transforms supporting frames to align with the reference frame. To predict the HR video frame, a reconstruction network taking aligned frames and the reference frame is utilized. Experimental results demonstrate that the TDAN is capable of alleviating occlusions and artifacts for temporal alignment and the TDAN-based VSR model outperforms several recent state-of-the-art VSR networks with a comparable or even much smaller model size. The source code and pre-trained models are released in https://github.com/YapengTian/TDAN-VSR.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/02c6f4bb3adb4ce080575c122eb6430d2dc2a8ef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Radu Timofte
4. Chao Dong
5. Lei Zhang
6. Kai Zhang
7. Ming-Hsuan Yang
8. Yu Qiao

Citations in the corpus

• EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “In particular, TDAN [40] uses deformable convolutions to align the input frames at the feature level without explicit motion estimation or image warping.”
• FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “[9] proposed to use deformable convolution together with the learnt offset maps to enhance the modeling capability of convolution neural networks, which has achieved promising results in several videorelated tasks like action recognition [18] and video superresolution [30, 35].”
• BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, background): “For example, TDAN [30] adopts deformable convolutions (DCNs) [5, 37] to align different frames at the feature level.”
• BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, methodology): “One prevalent approach is the sliding-window framework [9, 28, 31, 35], where each frame in the video is restored using the frames within a short temporal window.”
• Neural Video Compression with Diverse Contexts (2023, background): “The works [9,54,58] show deformable alignment gets better results for video restoration as each location has multiple offsets to capture temporal correspondence.”
• Real-World Single Image Super-Resolution: A Brief Review (2021, background): “On the whole, VSR aims to improve the spatial resolution (known as spatial VSR) [11]–[19] or the frame rate (known as temporal VSR) [20]–[27] of the observed video.”
• FaPN: Feature-aligned Pyramid Network for Dense Image Prediction (2021, neutral): “To establish accurate correspondences among multiple frames given a large motion for video restoration, TDAN [42] and EDVR [43] achieve implicit motion compensation with by deformable convolution [10] at the feature level.”
• Changer: Feature Interaction is What You Need for Change Detection (2022, methodology): “Moreover, alignment-based fusion methods focus on feature alignment, and they typically use flow field or de-formable convolution [10, 50] to align features of different levels in the spatial dimension [16,17,21,34,38].”
• VRT: A Video Restoration Transformer (2022, methodology): “2) Temporal Alignment and Fusion: Since supporting frames are often highly-related but misaligned, temporal alignment plays an critical role in video restoration [2], [3], [12], [13], [31].”