Pixel Recursive Super Resolution

By Ryan Dahl, Mohammad Norouzi, Jonathon Shlens. 2017

263 citations recorded by the source corpus.

Super resolution is the problem of artificially enlarging a low resolution photograph to recover a plausible high resolution version. In the regime of high magnification factors, the problem is dramatically underspecified and many plausible, high resolution images may match a given low resolution image. In particular, traditional super resolution techniques fail in this regime due to the multimodality of the problem and strong prior information that must be imposed on image synthesis to produce plausible high resolution images. In this work we propose a new probabilistic deep network architecture, a pixel recursive super resolution model, that is an extension of PixelCNNs to address this problem. We demonstrate that this model produces a diversity of plausible high resolution images at large magnification factors. Furthermore, in human evaluation studies we demonstrate how previous methods fail to fool human observers. However, high resolution images sampled from this probabilistic deep network do fool a naive human observer a significant fraction of the time.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

Image Transformer (2018, methodology): “Following the same procedure for the evaluation study as (Dahl et al., 2017), we show 50 pairs of images, selected randomly from the validation set, to 50 workers each.”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “Dahl et al. (2017) use a PixelCNN as a prior for image super-resolution with a convolutional neural network.”
The Perception-Distortion Tradeoff (2017, background): “-fake user studies, which examine the ability of human observers to tell whether x̂ is real or the output of an algorithm [48, 14, 6, 13, 49, 22, 10] (similarly to the idea underlying generative adversarial nets [9]).”
Palette: Image-to-Image Diffusion Models (2021, methodology): “We avoid pixel-level metrics like PSNR and SSIM as they are not reliable measures of sample quality for difficult tasks that require hallucination, like recent super-resolution work, where [Dahl et al. 2017; Ledig et al. 2017; Menon et al. 2020] observe that PSNR and SSIM tend to prefer blurry regre”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, methodology): “Of particular relevance to our work are approaches which combine autoregressive models with multiscale modelling [49, 9, 39, 35].”
Fast Generation for Convolutional Autoregressive Models (2017, background): “Production environments have tight latency constraints, so real-time speech generation, machine translation, and image super-resolution (Dahl et al., 2017) all require fast generation.”
The 2018 PIRM Challenge on Perceptual Image Super-Resolution (2018, unknown): “[18, 13, 24, 31, 4] (see Fig.”
Learning Enriched Features for Real Image Restoration and Enhancement (2020, background): “Other networks designed to perform SR include recursive learning [57,41,3], progressive reconstruction [105,60], dense connections [99,104,127], attention mechanisms [125,23,126], multi-branch learning [60,66,22,64], and generative adversarial networks (GANs) [104,76,87,63].”
2018 PIRM Challenge on Perceptual Image Super-resolution (2018, unknown): “That is, models which excel at minimizing the reconstruction error tend to produce visually unpleasing results, while models that produce results with superior visual quality are rated poorly by distortion measures like PSNR, SSIM, IFC, etc. [18, 13, 24, 31, 4] (see Fig.”
Detecting Overfitting of Deep Generative Networks via Latent Recovery (2019, background): “For example, if a user is using a neural net to inpaint faces as in [17] or to perform super-resolution enhancing [9], it seems necessary to ensure verbatim copies of training images do not appear, due to privacy or even copyright concerns.”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “To improve the perceptual quality of the generated HR images, generative-based SR models attract increasing attention, e.g. , autoregressive-based models [6, 26, 29, 30].”
Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution (2017, methodology): “[5] use PixelCNN [27] to synthesize realistic details.”
PixelCNN Models with Auxiliary Variables for Natural Image Modeling (2016, background): “Similar constructions were suggested in (van den Oord et al., 2016b) in the context of recurrent networks and (Dahl et al., 2017) for the problem of super-resolution.”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, background): “4 [13] Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens.”
PixColor: Pixel Recursive Colorization (2017, methodology): “Our architecture is based on [8] who used PixelCNNs to perform super resolution (another one-to-many problem).”
E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles (2019, background): “(We train super-resolution in a supervised fashion [13] as a case study, aware that state-of-the-art techniques utilize more sophisticated models [7, 18].)”
Controllable Semantic Image Inpainting (2018, background): “In order that the manipulation is intuitive, we desire latent variables that correspond to a disentangled and more interpretable semantic representation of the image [3, 11, 2, 16].”
Deep Generative Adversarial Compression Artifact Removal (2017, methodology): “Similar approaches have been devised, to target different image transformation problems, such as image superresolution [1, 17, 14, 3], style-transfer [10, 14] and image de-noising [40].”