Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
By Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi. 2016
12,248 citations recorded by the source corpus.
Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.
Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Media Technology
- Radu Timofte
- Luc Van Gool
- Lei Zhang
- Ming-Hsuan Yang
- Chao Dong
- Kai Zhang
- Wangmeng Zuo
- Chen Change Loy
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Stefano Ermon
- Alexei A. Efros
- Ming-Hsuan Yang
- Daniel Cohen-Or
- Arash Vahdat
Citations in the corpus
- Image Transformer (2018, background): “…to the often blurry images generated by networks trained with likelihood-based losses, generative adversarial networks (GANs) have been shown to produce sharper images with realistic high-frequency detail in generation and image super-resolution tasks (Zhang et al., 2016; Ledig et al., 2016).”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In particular, relative patch prediction [23, 40], colorizing gray-scale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful.”
- Neural Discrete Representation Learning (2017, background): “Recent advances in generative modelling of images [38, 12, 13, 22, 10], audio [37, 26] and videos [20, 11] have yielded impressive samples and applications [24, 18].”
- Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “The samples generated from these models are hard to distinguish from real data without close inspection, and their applications range from super resolution [21] to domain editing [44], artistic manipulation [36], or text-to-speech and music generation [25].”
- Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, methodology): “…using two pre-trained neural networks: a convolutional autoencoder trained to remove JPEG artifacts in natural images, similar in structure to the proposed by Mao et al. (2016a), and an adversarially-trained 4x super-resolution network (Korobchenko & Foco, 2017) similar to Ledig et al. (2016).”
- Generative Adversarial Networks (2021, neutral): “Different GAN architectures were proposed for different tasks, such as image super-resolution [38] and image-to-image transfer [79, 53].”
- Image Super-Resolution via Iterative Refinement (2021, background): “GAN-based super-resolution methods have also found considerable success [12], [16], [17], [30], [31], [32], [59].”
- Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, background): “Since the PSNR-oriented BSRNet model tends to produce oversmoothed results due to the pixel-wise average problem [24], the perceptual quality-oriented model is preferred for real applications [5].”
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “t images. Each track had approximately 20 algorithm submissions. We also evaluate several additional methods, including bicubic upsampling, and four of the top performing deep superresolution methods [24, 59, 31, 48]. A common qualitative way of presenting superresolution results is zooming into spe”
- Unrolled Generative Adversarial Networks (2016, background): “…functions, without explicitly computing likelihoods or performing inference, and while targeting more mode-seeking divergences as made GANs extremely successful in image generation (Odena et al., 2016; Salimans et al., 2016; Radford et al., 2015), and image super resolution (Ledig et al., 2016).”
- StyleGAN-NADA (2021, background): “These range from image enhancement [18,50], editing [13,36] and recently even discriminative tasks such as classification and regression [27, 48].”
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “Compared with other deep generative models, Generative Adversarial Networks (GANs) [6] have shown great performance for generating sharper samples [17, 3, 23, 13, 10, 35, 24, 34, 39, 40].”
- Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Although the features learned by generative models have been put to a wide variety of exciting uses [40, 69, 12, 43, 33], their power as latent representations for downstream visual tasks such as object recognition has yet to be fully realized.”
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “feature matching loss is related to the perceptual loss [11, 13,22], which has been shown to be useful for image superresolution [31] and style transfer [22].”