EDICT: Exact Diffusion Inversion via Coupled Transformations

By Bram Wallace, Akash Gokul, Nikhil Vijay Naik. Computer Vision and Pattern Recognition, 2022

308 citations recorded by the source corpus.

Finding an initial noise vector that produces an input image when fed into the diffusion process (known as inversion) is an important problem in denoising diffusion models (DDMs), with applications for real image editing. The standard approach for real image editing with inversion uses denoising diffusion implicit models (DDIMs [29]) to deterministically noise the image to the intermediate state along the path that the denoising would follow given the original conditioning. However, DDIM inversion for real images is unstable as it relies on local linearization assumptions, which result in the propagation of errors, leading to incorrect image reconstruction and loss of content. To alleviate these problems, we propose Exact Diffusion Inversion via Coupled Transformations (EDICT), an inversion method that draws inspiration from affine coupling layers. EDICT enables mathematically exact inversion of real and model-generated images by maintaining two coupled noise vectors which are used to invert each other in an alternating fashion. Using Stable Diffusion [25], a state-of-the-art latent diffusion model, we demonstrate that EDICT successfully reconstructs real images with high fidelity. On complex image datasets like MS-COCO, EDICT reconstruction significantly outperforms DDIM, improving the mean square error of reconstruction by a factor of two. Using noise vectors inverted from real images, EDICT enables a wide range of image edits—from local and global semantic edits to image stylization—while maintaining fidelity to the original image structure. EDICT requires no model training/finetuning, prompt tuning, or extra data and can be combined with any pretrained DDM.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/7e80f79472d9b5aaa109075910d3ff9f9149f4c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, background): “EDICT [22] suggests an image inversion based on two noise vectors enabling better image reconstruction and textual faithfulness.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Diffusion-based image editing Diffusion models have also shown great potential for semantic editing [33, 3, 2, 31, 79, 37, 72, 68, 63, 42, 43, 5, 71].”
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing (2023, background): “For this purpose, we assume the generation process of the original image is known, or can be derived via inversion [13, 27] for real images.”
Video-P2P: Video Editing with Cross-Attention Control (2023, unknown): “It is just an approximation because errors are accumulated in the denoising pass, consistent with conclusions in [24, 43].”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, unknown): “Similar scheme is also used in (Wallace et al., 2022).”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “However, due to the local linearization assumptions, DDIM [125] may lead to incorrect image reconstruction with the error propagation [130].”
An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, methodology): “[7, 17, 25, 4, 27, 19]) rely on an approximate inversion method for the denoising diffusion implicit model (DDIM) scheme [24], which is a deterministic sampling process that maps a single initial noise vector into a generated image.”