EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis

By Mehdi S. M. Sajjadi, Bernhard Schölkopf, Michael Hirsch. 2016

1,047 citations recorded by the source corpus.

Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack highfrequency textures and do not look natural despite yielding high PSNR values.,,We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixelaccurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

Projected GANs Converge Faster (2021, neutral): “Commonly, the losses are combined additively [10, 14, 32, 52, 64].”
Image Super-Resolution via Iterative Refinement (2021, methodology): “Following [31], [64] we apply 4 super-resolution models to 56 56 center crops from the validation set of ImageNet.”
Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “As mentioned in the introduction, existing DNNs-based SISR methods are generally based on bicubic downsam-pling [23, 44] and traditional degradations [26, 37, 54, 59, 60], or some simple variants [11, 41, 53, 56, 58].”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “Image generation tasks usually adopt style loss which poses as an effective tool to combat “checkerboard” artifacts [55].”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “[40] to be an effective tool to combat “checkerboard” artifacts caused by transpose convolution layers [37].”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “We also evaluate several additional methods, including bicubic upsampling, and four of the top performing deep superresolution methods [24, 58, 31, 47].”
Learning-based Video Motion Magnification (2018, methodology): “Our encoder and decoder are fully convolutional, which enables it to work on any resolution [18], and use residual blocks to generate high-quality output [19].”
The Perception-Distortion Tradeoff (2017, methodology): “This gives theoretical support to the growing empirical evidence of the advantages of GANs in image restoration [20, 34, 31, 46, 32, 14, 50].”
Deep Image Prior (2017, methodology): “GAN-based [11] methods SRGAN [19] and EnhanceNet [28] (not shown in the comparison) intelligently hallucinate fine details of the image, which is impossible with our method that uses absolutely no information about the world of HR images.”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, methodology): “r than merely comparing the appearance. Ledig et al. [1] propose SRGAN model that uses perceptual loss and adversarial loss to favor outputs residing on the manifold of natural images. Sajjadi et al. [16] develop a similar approach and further explored the local texture matching loss. Based on these”
Image Super-Resolution Using Very Deep Residual Channel Attention Networks (2018, methodology): “Evaluation Bicubic DRCN FSRCNN PSyCo ENet-E RCAN Baseline [19] [2] [45] [8] (ours)”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Since our method is able to generate the full-frame output, we can incorporate perceptual loss functions [11, 22, 27, 42, 65] to further improve the visual quality of the interpolation results, as shown in Figure 1.”
Second-Order Attention Network for Single Image Super-Resolution (2019, methodology): “Some loss functions have been widely used, such as L 2 [2, 12, 29, 30], L 1 [14, 15, 20, 39], perceptual losses [11, 26].”
DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks (2017, neutral): “However, those networks are trained on limited datasets and in real-world settings images are often degraded by different artifacts, including motion blur, Similar to [21] and [32] we studied the influence of motion blur on object detection and propose a new way to evaluate the quality of deblurring”
Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network (2018, methodology): “ EDSR network using a combination of loss functions that can improve distortion measures as well as perceptual quality. Motivated by the observations in Enhanced Perceptual Super-resolution Network 3 [22,28,39,5], we use a combination of MSE loss, perceptual (VGG) loss, and adversarial loss to train”
EdgeConnect: Structure Guided Image Inpainting using Edge Prediction (2019, methodology): “[26], which has achieved impressive results for style transfer, super-resolution [42, 18], and image-to-image translation [60].”