Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

By Jose Caballero, Christian Ledig, Andrew P. Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, Wenzhe Shi. Computer Vision and Pattern Recognition, 2016

740 citations recorded by the source corpus.

Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video super-resolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is end-to-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to single-frame models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-the-art performance in both accuracy and efficiency.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/0584f25256c83f10349d6108fde1bccd82908d8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network (2017, methodology): “We can use an approximate motion compensation [5] to align two frames, but then the evaluation system will have an intrinsic motion compensation noise which is not separable from the generative modeling noise.”
Blind image super-resolution with spatially variant degradations (2019, methodology): “In the context of video super-resolution, a standard approach is to rely on consecutive frames to achieve better results [Caballero et al. 2017; Sajjadi et al. 2018].”
Deep Learning for Image Super-Resolution: A Survey (2019, methodology): “…SR GAN EDSR [33] enhanced deep SR network SRDenseNet [79] SR DenseNet EDSR-PP [113] EDSR with pyramid pooling STCN [192] spatial-temporal CNN ESPCN [188] efﬁcient sub-pixel CNN TDAE [167] transformative discriminative auto-encoder ESRGAN [101] enhanced SRGAN TDN [165] transformative discriminative…”
FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation (2019, methodology): “Similarly to both DVDnet and ViDeNN, the use of spatio-temporal CNN blocks in restoration tasks has been also featured in [46, 7].”
Detail-Revealing Deep Video Super-Resolution (2017, methodology): “[2] proposed the first end-to-end video SR framework, which incorporates motion compensation as a submodule.”
Efficient Long-Range Attention Network for Image Super-resolution (2022, background): “In recent years, deep convolutional neural network (CNN) [24] based SR models [11, 12, 48, 30, 6, 51, 65, 47, 55] have become prevalent for their strong capability in recovering or generating [25, 56] image high-frequency details, showing high practical value in image and video restoration, transiti”
Frame-Recurrent Video Super-Resolution (2018, methodology): “While a number of the above mentioned methods are endto-end trainable, the authors often note that they first pre-train each component before fine-tuning the system as a whole in a final step [3, 28, 39].”
MBLLEN: Low-Light Image/Video Enhancement Using CNNs (2018, methodology): “Powerful tools such as end-to-end networks and GANs [13] have been employed by various applications, including image super resolution [4, 23], image denoising [31, 44] and image-to-image translation [16, 45].”
VRT: A Video Restoration Transformer (2022, methodology): “Existing video restoration methods can be mainly divided into two categories: sliding window-based methods [1], [2], [3], [4], [5], [6] and recurrent methods [7], [8], [9], [10], [11], [12], [13], [14], [15], [16].”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “…examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical ﬂow prediction [32, 47, 52], optical ﬂow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement [7, 46, 51].”
Image Super-Resolution by Neural Texture Transfer (2019, background): “The references could be selected from adjacent frames in a video [26, 2], images from web retrieval [39], an external database (dictionary) [42], or images from different view points [41].”
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, methodology): “[3] introduced the ﬁrst end-to-end VSR network: VESCPN, which jointly trains ﬂow estimation and spatiotemporal networks.”
Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (2019, methodology): “It is observed that VSR-NLr1 surpasses the base model VSR (about 0.14 dB), which is superior to VSR-STMC and VSR-SPMC.”
Video Enhancement with Task-Oriented Flow (2017, methodology): “Such approaches have been applied to video interpolation [18], light-field video interpolation [31], object novel view synthesis [37], eye gaze manipulation [7], and superresolution [5, 29].”
Recurrent Back-Projection Network for Video Super-Resolution (2019, methodology): “The frames can be also aligned explicitly, using motion cues between temporal frames with the alignment modules [25, 2, 30, 27].”
Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation (2018, methodology): “Upscale Metric Bicubic VSRnet[16] VESPCN[1] Tao et al.”
Artificial intelligence in the creative industries: a review (2020, methodology): “Later work exploited temporal correlation via a motion compensation process before concatenating multiple warped frames into a 3D volume [95] using a recurrent architecture [90].”