Frame-Recurrent Video Super-Resolution

By Mehdi S. M. Sajjadi, Raviteja Vemulapalli, Matthew A. Brown. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

570 citations recorded by the source corpus.

Recent advances in video super-resolution have shown that convolutional neural networks combined with motion compensation are able to merge information from multiple low-resolution (LR) frames to generate high-quality images. Current state-of-the-art methods process a batch of LR frames to generate a single high-resolution (HR) frame and run this scheme in a sliding window fashion over the entire video, effectively treating the problem as a large number of separate multi-frame super-resolution tasks. This approach has two main weaknesses: 1) Each input frame is processed and warped multiple times, increasing the computational cost, and 2) each output frame is estimated independently conditioned on the input frames, limiting the system's ability to produce temporally consistent results. In this work, we propose an end-to-end trainable frame-recurrent video super-resolution framework that uses the previously inferred HR estimate to super-resolve the subsequent frame. This naturally encourages temporally consistent results and reduces the computational cost by warping only one image in each step. Furthermore, due to its recurrent nature, the proposed method has the ability to assimilate a large number of previous frames without increased computational demands. Extensive evaluations and comparisons with previous methods validate the strengths of our approach and demonstrate that the proposed framework is able to significantly outperform the current state of the art.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/aa3f7330f9b27e57745c4d14753893964ad91d5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Radu Timofte
3. Chen Change Loy
4. Lei Zhang
5. Ming-Hsuan Yang
6. Chao Dong
7. Dacheng Tao
8. Yu Qiao

In Media Technology

1. Radu Timofte
2. Chao Dong
3. Luc Van Gool
4. Kai Zhang
5. Lei Zhang
6. Yulun Zhang
7. Chen Change Loy
8. Wangmeng Zuo

Citations in the corpus

• TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, methodology): “[22] extended the conventional VSR models to a frame-recurrent VSR framework.”
• Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (2019, methodology): “Because Vid4 dataset only contains 4 scenes under low resolution (smaller than 720 × 576), we further collect 10 frame sequences (1272 × 720) from [26, 39] for testing.”
• BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, unknown): “Most existing methods adopt unidirectional propagation [12,14,24].”
• FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “In many downstream video tasks, such as action recognition [46,37,61], video inpainting [28,51,13], video super-resolution [30,5,39], and frame interpolation [52,33,20], optical flow serves as a fundamental component providing dense correspondences as valuable clues for prediction.”
• BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, background): “More examples are provided in the appendix.”
• NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study (2019, methodology): “For video super-resolution, vid4 dataset [21] with 155 frames is used for comparison and each work employs different training datasets [26, 9, 29, 3].”
• Recurrent Back-Projection Network for Video Super-Resolution (2019, methodology): “In RNNs [30, 27, 13], it is not easy to jointly model subtle and significant changes (e.”
• No More Strided Convolutions or Pooling: A New CNN Building Block for Low-Resolution Images and Small Objects (2022, unknown): “Our SPD component generalizes a (raw) image transformation technique [29] to downsampling feature maps inside and throughout a CNN, as follows.”
• Light Field Super-Resolution with Zero-Shot Learning (2021, unknown): “, disparity [12] and optical flow [31].”
• VRT: A Video Restoration Transformer (2022, methodology): “Existing video restoration methods can be mainly divided into two categories: sliding window-based methods [1], [2], [3], [4], [5], [6] and recurrent methods [7], [8], [9], [10], [11], [12], [13], [14], [15], [16].”
• Blind image super-resolution with spatially variant degradations (2019, methodology): “In the context of video super-resolution, a standard approach is to rely on consecutive frames to achieve better results [Caballero et al. 2017; Sajjadi et al. 2018].”
• Deep Learning for Image Super-Resolution: A Survey (2019, methodology): “And FRVSR [194] uses previously inferred HR estimates to reconstruct the subsequent HR frame by two deep CNNs in a recurrent manner.”
• Non-Discriminative Data or Weak Model? On the Relative Importance of Data and Model Resolution (2019, methodology): “An input image of an arbitrary resolution is fed into the first network layer by using the space-to-depth [10, 11, 12] operator or image rescaling.”
• Efficient Long-Range Attention Network for Image Super-resolution (2022, background): “In recent years, deep convolutional neural network (CNN) [23] based SR models [11, 12, 47, 29, 6, 50, 60, 46, 53] have become prevalent for their strong capability in recovering or generating [24, 54] image high-frequency details, showing high practical value in image and video restoration,…”
• EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “We compare our EDVR method with nine algorithms: RCAN [51], DeepSR [19], BayesSR [21], VESPCN [2], SPMC [37], TOFlow [48], FRVSR [32], DUF [10] and RBPN [6] on three testing datasets: Vid4 [21], Vimeo-90K-T [48] and REDS4.”