ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting

By Zongsheng Yue, Jianyi Wang, Chen Change Loy. Neural Information Processing Systems, 2023

569 citations recorded by the source corpus.

Diffusion-based image super-resolution (SR) methods are mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose a novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post-acceleration during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, even only with 15 sampling steps. Our code and model are available at https://github.com/zsyOAOA/ResShift.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d4a60ef37125fcde198781c2eb578a9c9dc78c1c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Yu Qiao
2. Wenhan Yang
3. Ziwei Liu
4. Lei Zhang
5. Wangmeng Zuo
6. David Dapeng Zhang
7. Xiangchu Feng
8. Kai Zhang

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Luc Van Gool
3. Lei Zhang
4. Chen Change Loy
5. Ming-Hsuan Yang
6. Ziwei Liu
7. Dacheng Tao
8. Xiaogang Wang

Citations in the corpus

• SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “Experimental results demonstrate that our method achieves comparable or even better performance compared to SOTA methods and the teacher diffusion model [46], while greatly reducing the number of inference steps from 15 to 1 , resulting in up to a × 10 speedup in inference.”
• Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, unknown): “Fast sampling strategy and model distillation are two promising solutions to improve efficiency, and such exploration is left as our future work.”
• SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, unknown): “Datasets Metrics BSRGAN [70] Real-[57] ESRGAN LDL [34] DASR [35] FeMaSR [5] LDM [46] StableSR [55] ResShift [66] PASD [65]”
• Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “By introducing a pixel-aware cross attention module, PASD succeeded in perceiving image local structures in pixel-level and achieved robust and perceptually realistic Real-ISR results.”