Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization

By Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang. European Conference on Computer Vision, 2023

326 citations recorded by the source corpus.

Diffusion models have demonstrated impressive performance in various image generation, editing, enhancement and translation tasks. In particular, the pre-trained text-to-image stable diffusion models provide a potential solution to the challenging realistic image super-resolution (Real-ISR) and image stylization problems with their strong generative priors. However, the existing methods along this line often fail to keep faithful pixel-wise image structures. If extra skip connections between the encoder and the decoder of a VAE are used to reproduce details, additional training in image space will be required, limiting the application to tasks in latent space such as image stylization. In this work, we propose a pixel-aware stable diffusion (PASD) network to achieve robust Real-ISR and personalized image stylization. Specifically, a pixel-aware cross attention module is introduced to enable diffusion models perceiving image local structures in pixel-wise level, while a degradation removal module is used to extract degradation insensitive features to guide the diffusion process together with image high level information. An adjustable noise schedule is introduced to further improve the image restoration results. By simply replacing the base diffusion model with a stylized one, PASD can generate diverse stylized images without collecting pairwise training data, and by shifting the base model with an aesthetic one, PASD can bring old photos back to life. Extensive experiments in a variety of image enhancement and stylization tasks demonstrate the effectiveness of our proposed PASD approach. Our source codes are available at \url{https://github.com/yangxy/PASD/}.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/ccf43c35160954616b8f1c3c00e939883b666c2f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, unknown): “To incorporate soft prompts into the diffusion process, we adopt the cross-attention mechanism proposed in PASD [65] to learn semantic guidance.”
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, methodology): “Low-Quality Input Real-ESRGAN+[78] StableSR[75] DiffBIR[49] PASD[85] SUPIR (ours) SUPIR (ours).”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “PASD [66] has proposed a PACA module, which could effectively inject the pixel-level condition information into diffusion prior and achieve higher fidelity.”