Recurrent Back-Projection Network for Video Super-Resolution
By Muhammad Haris, Gregory Shakhnarovich, Norimichi Ukita. Computer Vision and Pattern Recognition, 2019
498 citations recorded by the source corpus.
We proposed a novel architecture for the problem of video super-resolution. We integrate spatial and temporal contexts from continuous video frames using a recurrent encoder-decoder module, that fuses multi-frame information with the more traditional, single frame super-resolution path for the target frame. In contrast to most prior work where frames are pooled together by stacking or warping, our model, the Recurrent Back-Projection Network (RBPN) treats each context frame as a separate source of information. These sources are combined in an iterative refinement framework inspired by the idea of back-projection in multiple-image super-resolution. This is aided by explicitly representing estimated inter-frame motion with respect to the target, rather than explicitly aligning frames. We propose a new video super-resolution benchmark, allowing evaluation at a larger scale and considering videos in different motion regimes. Experimental results demonstrate that our RBPN is superior to existing methods on several datasets.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/bcba77a81a59dcf0798e538e513d8b15b229634f
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Radu Timofte
- Chen Change Loy
- Chao Dong
- Lei Zhang
- Kai Zhang
- Ming-Hsuan Yang
- Yun Raymond Fu
Citations in the corpus
- Artificial intelligence in the creative industries: a review (2020, methodology): “The original target frame is fed, along with its neighbouring frames, into intermediate layers of the CNN to perform inter-frame motion compensation during feature extraction [103].”
- VRT: A Video Restoration Transformer (2022, methodology): “Existing video restoration methods can be mainly divided into two categories: sliding window-based methods [1], [2], [3], [4], [5], [6] and recurrent methods [7], [8], [9], [10], [11], [12], [13], [14], [15], [16].”
- EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “We compare our EDVR method with nine algorithms: RCAN [51], DeepSR [19], BayesSR [21], VESPCN [2], SPMC [37], TOFlow [48], FRVSR [32], DUF [10] and RBPN [6] on three testing datasets: Vid4 [21], Vimeo-90K-T [48] and REDS4.”
- BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, methodology): “pixel-shuffle [27]), BasicVSR outperforms existing state of the arts [9, 12, 32] in both performance (up to 0.”
- TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, background): “Recently, RBPN [37] has shown promising VSR results.”
- BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, background): “One prevalent approach is the sliding-window framework [9, 28, 31, 35], where each frame in the video is restored using the frames within a short temporal window.”