SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models

By Haoying Li, Meng Chang, Huajun Feng, Zhi-hai Xu, Qi Li, Yue-ting Chen. 2021

889 citations recorded by the source corpus.

Single image super-resolution (SISR) aims to reconstruct high-resolution (HR) images from the given low-resolution (LR) ones, which is an ill-posed problem because one LR image corresponds to multiple HR images. Recently, learning-based SISR methods have greatly outperformed traditional ones, while suffering from over-smoothing, mode collapse or large model footprint issues for PSNR-oriented, GAN-driven and flow-based methods respectively. To solve these problems, we propose a novel single image super-resolution diffusion probabilistic model (SRDiff), which is the first diffusion-based model for SISR. SRDiff is optimized with a variant of the variational bound on the data likelihood and can provide diverse and realistic SR predictions by gradually transforming the Gaussian noise into a super-resolution (SR) image conditioned on an LR input through a Markov chain. In addition, we introduce residual prediction to the whole framework to speed up convergence. Our extensive experiments on facial and general benchmarks (CelebA and DIV2K datasets) show that 1) SRDiff can generate diverse SR results in rich details with state-of-the-art performance, given only one LR input; 2) SRDiff is easy to train with a small footprint; and 3) SRDiff can perform flexible image manipulation including latent space interpolation and content fusion.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

On Fast Sampling of Diffusion Probabilistic Models (2021, background): “…2020; Dhariwal and Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020; Okamoto et al., 2021), image or audio super-resolution (Li et al., 2021; Lee and Han, 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), music synthesis (Liu et al., 2021; Mittal et al.,…”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “, 2021), image super-resolution (Saharia et al., 2021; Li et al., 2021), image-to-image translation (Sasaki et al.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Unlike in other conditional DPMs [23, 32, 45] that use spatial conditional variables (e.”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…including unconditional image generation [Song and Ermon, 2019, Ho et al., 2020, Song et al., 2021, Nichol and Dhariwal, 2021], shape generation [Cai et al., 2020], text-to-speech [Chen et al., 2021, Kong et al., 2020] and single image super-resolution [Saharia et al., 2021, Li et al., 2021].”
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, background): “, 2021), image super-resolution (Saharia et al., 2021; Li et al., 2021), image-to-image translation (Sasaki et al.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “To further enhance the spatial resolution, we also temporally align pixel-space and latent DM upsamplers [29], which are widely used for image super resolution [43, 65, 68, 69], turning them into temporally consistent video super resolution models.”
SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, background): “As far as we know, with the exception of super resolution [19,27,41], diffusion models have not been applied to problems in which the ground truth result is unique.”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, methodology): “A prevalent approach is to concatenate the LR image with the noise in each step and retrain the diffusion model from scratch [10, 11, 45].”
Implicit Diffusion Models for Continuous Super-Resolution (2023, positive): “3, unlike the cross-attention mechanism [34] and the concatenating operation [20], we map s to a set of scaling vectors α = α (1) 2 , . . . , α 2 , . . . , α with an adaptive MLP, where i represents the depth index with different resolution outputs from the conditioning network and the denoising…”
Q-Diffusion: Quantizing Diffusion Models (2023, background): “As a class of flexible generative models, diffusion models demonstrate their power in various applications such as image super-resolution [37, 18], inpainting [44], shape generation [3], graph generation [31], image-to-image translation [40], and molecular conformation generation [47].”
DDGR: Continual Learning with Deep Diffusion-based Generative Replay (2023, background): “Recently, diffusion models have shown their exquisite potential in the field of computer vision (Dhariwal & Nichol, 2021; Ho et al., 2022b; Li et al., 2022; Luo & Hu, 2021), audio processing (Chen et al., 2021; Popov et al., 2021; Kim et al., 2022), and AI for science (Luo et al., 2021; Xu et al.,…”
Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction (2021, background): “Interestingly, diffusion models can go beyond unconditional image synthesis, and have been applied to conditional image generation, including super-resolution [5, 17, 25], inpainting [31, 34], MRI reconstruction [6, 13, 33], image translation [5, 19, 28], and so on.”
Image Restoration with Mean-Reverting Stochastic Differential Equations (2023, methodology): “Among the commonly used formulations (Yang et al., 2022), we adopt that of using the diffusion models defined via stochastic differential equations (SDEs, Song et al., 2021b;c).”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “Moreover, training a network from scratch can potentially jeopardize the generative priors captured in synthesis models, leading to suboptimal performance in the ﬁnal network.”
TabDDPM: Modelling Tabular Data with Diffusion Models (2022, background): “…the advantages of diffusion models are successfully exploited in applications, such as colorization (Song et al., 2021), in-painting (Song et al., 2021), segmentation (Baranchuk et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021), semantic editing (Meng et al., 2021) and others.”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “To date, diffusion models have been found to be useful in a wide variety of areas, ranging from generative modeling tasks such as image generation [11], image super-resolution [12], image inpainting [13] to discriminative tasks such as image segmentation [14], classification [15], and anomaly detect”
Label-Efficient Semantic Segmentation with Diffusion Models (2021, result): “…of DDPM are successfully exploited in applications, such as colorization (Song et al., 2021), inpainting (Song et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021b), and semantic editing (Meng et al., 2021), where DDPM often achieve more impressive results compared to GANs.”
Deblurring via Stochastic Refinement (2021, methodology): “Likelihood-based deep generative models such as Variational Autoencoders [53], Normalizing Flows [44,45], and Diffusion Probabilistic Models (DPMs) [40,58] have also been successfully applied to other image enhancement tasks such as super-resolution, where a diverse set of candidates can be…”
VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation (2023, methodology): “Based on this formulation, DPM has been applied to various generative tasks, such as image-generation [15, 25], super-resolution [19, 28], image translation [31], etc ., and become an important class of deep generative models.”