Image Super-Resolution via Iterative Refinement

By Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, Mohammad Norouzi. 2021

2,574 citations recorded by the source corpus.

We present SR3, an approach to image Super-Resolution via Repeated Refinement. SR3 adapts denoising diffusion probabilistic models (Ho et al. 2020), (Sohl-Dickstein et al. 2015) to image-to-image translation, and performs super-resolution through a stochastic iterative denoising process. Output images are initialized with pure Gaussian noise and iteratively refined using a U-Net architecture that is trained on denoising at various noise levels, conditioned on a low-resolution input image. SR3 exhibits strong performance on super-resolution tasks at different magnification factors, on faces and natural images. We conduct human evaluation on a standard 8× face super-resolution task on CelebA-HQ for which SR3 achieves a fool rate close to 50%, suggesting photo-realistic outputs, while GAN baselines do not exceed a fool rate of 34%. We evaluate SR3 on a 4× super-resolution task on ImageNet, where SR3 outperforms baselines in human evaluation and classification accuracy of a ResNet-50 classifier trained on high-resolution images. We further show the effectiveness of SR3 in cascaded image generation, where a generative model is chained with super-resolution models to synthesize high-resolution images with competitive FID scores on the class-conditional 256×256 ImageNet generation challenge.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “In the context of image synthesis, however, combining the generative power of DMs with other types of conditionings beyond class-labels [15] or blurred variants of the input image [70] is so far an under-explored area of research.”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “For the first upsampling stage, we use gaussian blur [43], and for the second, we use a more diverse BSR degradation [42,”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “…x t ∼ q ( x t | x 0 ) by applying Gaussian noise (cid:15) to to x 0 , then train a model (cid:15) θ to predict the added noise using a standard mean-squared error loss: Diffusion models have also been successfully applied to image super-resolution (Nichol & Dhariwal, 2021; Saharia et al., 2021b).”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, methodology): “4), and they use architectures similar to SR3 [21].”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “In this section, we demonstrate that we can achieve results better than StyleGAN2 and BigGAN-deep with the same or lower compute budget.”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “, 2021), image super-resolution (Saharia et al., 2021; Li et al., 2021), image-to-image translation (Sasaki et al.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Numerous applications have emerged notably in the image domain, such as image manipulation, translation, super-resolution [8,32,35,43], in speech and text domains [5,6], or 3D point cloud [34].”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, background): “Diffusion generative models were first proposed in [55] and achieved strong results on image generation [12,20,21,35] and image super super-resolution [48] recently.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “Diffusion models have seen wide success in image generation [28, 42, 62, 16, 29, 61], outperforming GANs in fidelity and diversity, without training instability and mode collapse issues [6, 16, 29].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, positive): “This idea is in line with the latest work on diffusion models [Dhariwal and Nichol 2021; Ho et al. 2022; Nichol and Dhariwal 2021; Saharia et al. 2021].”
Flow Matching for Generative Modeling (2022, neutral): “FM-OT achieves similar PSNR and SSIM values to (Saharia et al., 2022) while considerably improving on FID and IS, which as argued by (Saharia et al.”
Palette: Image-to-Image Diffusion Models (2021, methodology): “Conditional diffusion models [Chen et al. 2021a; Saharia et al. 2021] make the denoising process conditional on an input signal.”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…including unconditional image generation [Song and Ermon, 2019, Ho et al., 2020, Song et al., 2021, Nichol and Dhariwal, 2021], shape generation [Cai et al., 2020], text-to-speech [Chen et al., 2021, Kong et al., 2020] and single image super-resolution [Saharia et al., 2021, Li et al., 2021].”
Gotta Go Fast When Generating Data with Score-Based Models (2021, methodology): “They have further been used effectively for super-resolution [Saharia et al., 2021, Kadkhodaie and Simoncelli, 2020], inpainting [Kadkhodaie and Simoncelli, 2020, Song and Ermon, 2020], source separation [Jayaram and Thickstun, 2020], and image-to-image translation [Sasaki et al., 2021].”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “This line of work has been successfully applied to modeling high dimensional natural images (Dhariwal & Nichol, 2021; Saharia et al., 2021), audio (Kong et al.”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “Images from the batch generated with 1536 latents (Figure 7) upsampled to 256 × 256 pixels using SR3 (Saharia et al., 2021). changes or adding compute requirements for layers other than the cross-attend.”
BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models (2022, methodology): “Therefore, most of the conditional diffusion models suffer from poor model generalization, and can only be adapted to some specific applications where the conditional input has high similarity with the output, such as inpainting and super-resolution [2, 4, 30].”