RePaint: Inpainting using Denoising Diffusion Probabilistic Models

By Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, Luc Van Gool. Computer Vision and Pattern Recognition, 2022

2,174 citations recorded by the source corpus.

Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. Most existing approaches train for a certain distribution of masks, which limits their generalization capabilities to unseen mask types. Furthermore, training with pixel-wise and perceptual losses often leads to simple textural extensions towards the missing areas instead of semantically meaningful generation. In this work, we propose RePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pretrained unconditional DDPM as the generative prior. To condition the generation process, we only alter the reverse diffusion iterations by sampling the unmasked regions using the given image infor-mation. Since this technique does not modify or condition the original DDPM network itself, the model produces high-quality and diverse output images for any inpainting form. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks. Re-Paint outperforms state-of-the-art Autoregressive, and GAN approaches for at least five out of six mask distributions. Github Repository: git.io/RePaint

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1e91fa21b890a8f5d615578f4ddf46c3cb394691

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, neutral): “, inpainting, colorization) [34], [35], [36].”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, neutral): “Generative inpainting systems such as LaMa [54], Latent Diffusion inpainting [42], RePaint [33], Stable Diffusion Inpainting variants 2 , and FLUX.1 Fill 3 make such context-aware edits instantaneous; see also Palette [45] and Paint-by-Example [57].”
An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, background): “Recent works harness diffusion models for various image editing and manipulation tasks, including text-guided editing [7, 4, 25, 1, 12], inpainting [15], and image-to-image translation [22, 16, 28].”
Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model (2022, background): “, 2021), RePaint(Lugmayr et al., 2022) and DDRM(Kawar et al.”
Denoising Diffusion Models for Plug-and-Play Image Restoration (2023, methodology): “, [7,8,38] and sampling methods in Appendix A.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Diffusion models can be directly applied to various inverse problems, such as superresolution [13, 64], inpainting [11, 47], deblurring [35, 82], and JPEG restoration [36, 62].”
Diffusion Models already have a Semantic Latent Space (2022, background): “The other works try to control the generative process using image-space guidance (Choi et al. (2021), Meng et al. (2021), Lugmayr et al. (2022), Avrahami et al. (2022)).”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “To date, diffusion models have been found to be useful in a wide variety of areas, ranging from generative modeling tasks such as image generation [11], image super-resolution [12], image inpainting [13] to discriminative tasks such as image segmentation [14], classification [15], and anomaly detect”
NADiffuSE: Noise-aware Diffusion-based Model for Speech Enhancement (2023, neutral): “Many works [18]–[20] have employed the DDPM to generate natural and high-quality human voice from the given text or Mel spectrogram.”
EDGE: Editable Dance Generation From Music (2022, methodology): “To enable editing for dances generated by EDGE, we use the standard masked denoising technique from diffusion image inpainting [33].”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “…(Ours) LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ours) LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ours) LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ours) HQ LQ ICT [14] LaMa [12] RePaint [8] DiffIR S 2 (Ours) Figure 2.”
Effective Data Augmentation With Diffusion Models (2023, neutral): “Inpainting with diffusion is one such approach that allows the user to specify what to edit as a mask Saharia et al. (2022a); Lugmayr et al. (2022).”
Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (2022, methodology): “Conditional generative modeling with diffusion models: Diffusion models have been used for related tasks such as inpainting, in particular in the image domain [41, 32].”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, positive): “Zero-Shot Editing Some methods [34, 8, 4, 31, 14] use diffusion-based generate models to achieve zero-shot editing.”