Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
By Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, Zehan Wang. 2016
6,030 citations recorded by the source corpus.
Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.
Subfields: Computer Vision and Pattern Recognition, Media Technology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
- Chen Change Loy
- Trevor Darrell
- Jitendra Malik
- Yi Yang
In Media Technology
Citations in the corpus
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, methodology): “Table 1: Comparison of methods: bicubic, SRCNN [7], SelfExSR [24], DRCN [27], ESPCN [40] on Set5, Set14, BSD100 benchmark data.”
- Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “Rather than straight-forward transposed convolutions, our STE model also uses the more efficient pixel shuffling operation [39].”
- Parallel Multiscale Autoregressive Density Estimation (2017, background): “Shi et al. (2016) developed a sub-pixel convolutional network well-suited to this problem.”
- Amortised MAP Inference for Image Super-resolution (2016, methodology): “The loss function is typically a simple convex function most often MSE `MSE(y, ŷ) = ‖y − ŷ‖22 as in (Dong et al., 2016; Shi et al., 2016).”
- A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Figure 11: The proposed efficient sub-pixel convolutional neural network (ESPCN) [53], with two convolution layers for feature maps extraction, and a sub-pixel convolution layer that aggregates the feature maps from Low Resolution space and builds the Super Resolution image in a single step.”
- Recent advances in convolutional neural networks (2015, methodology): “P ∈R After obtaining the distribution P ( p 1 , ..., p |R j | ) , we can sample from the multi-nomial distribution based on p to pick a location l within the region, then set the pooled activation as y j = a l , where l ∼ P ( p 1 , ..., p |R j | ) .”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “Those applications have driven the growing demand to deploy GANs, usually their trained generators , on resource-constrained platforms, e.g., for real-time style transfer and super resolution (Shi et al., 2016) in mobile APPs.”
- Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, methodology): “Moreover, we also consider 2 trainable convolutional layers, namely the deconvolution layer (also known as transposed convolution) [24] and the sub-pixel convolution [25].”
- SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, methodology): “By using a spatial broadcast network (Watters et al., 2019) which is much weaker when compared to other decoders like sub-pixel convolutional nets (Shi et al. (2016)), we limit the capacity of background module, which favors modeling static objects as foreground rather than background.”
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, methodology): “only fully-connected networks, while the CONV-SQAIR replaces the networks used to encode images and glimpses with convolutional ones; it also uses a subpixel-convolution network as the glimpse decoder (Shi et al., 2016).”
- Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Traditionally, image super-resolution has used dictionary-style methods [7, 49], matching patches of images to higher-resolution counterparts.”
- Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding (2019, methodology): “We describe our component model derived from Kankanahalli-Net that consists of bottleneck residual learning [24], soft-to-hard quantization [25], and sub-pixel convolutional neural networks for upsampling [26].”
- End-To-End Optimized Speech Coding with Deep Neural Networks (2017, methodology): “All convolutions use 1D filters of size 9 and PReLU activations [12]; the upsample block uses subpixel convolutions [13].”
- i-RevNet: Deep Invertible Networks (2018, methodology): “We use the inverse of the operation described in Shi et al. (2016) as illustrated in Figure 2, since it preserves roughly the spatial ordering, and thus permits to avoid mixing different neighborhoods via the next convolution.”
- Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, methodology): “In the auxiliary decoder and the modulator the upsampling (by a factor of 2) is done in the beginning using subpixel convolutional layer (Shi et al., 2016 For the first level model, which maps 256 × 256 RGB images to 64 × 64 3-channel codes with 8 bits, we use the same model as described in Section…”
- EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis (2016, methodology): “They further include dictionary-based methods [33, 40, 52, 59, 61, 64] that learn a sparse representation of image patches as a combination of dictionary atoms, as well as neural network-based approaches [4, 8, 9, 24, 25, 26, 47, 48, 62] which apply convolutional neural networks (CNNs) to the task o”
- Lossy Image Compression with Compressive Autoencoders (2017, methodology): “Inspired by the work of Shi et al. (2016), most convolutions are performed in a downsampled space to speed up computation, and upsampling is performed using sub-pixel convolutions (convolutions followed by reshaping/reshuffling of the coefficients).”