Paint by Example: Exemplar-based Image Editing with Diffusion Models

By Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, Fang Wen. Computer Vision and Pattern Recognition, 2022

643 citations recorded by the source corpus.

Language-guided image editing has achieved great success recently. In this paper, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. However, the naive approach will cause obvious fusing artifacts. We carefully analyze it and propose a content bottleneck and strong augmentations to avoid the trivial solution of directly copying and pasting the exemplar image. Meanwhile, to ensure the controllability of the editing process, we design an arbitrary shape mask for the exemplar image and leverage the classifier-free guidance to increase the similarity to the exemplar image. The whole framework involves a single forward of the diffusion model without any iterative optimization. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. The code and pretrained models are available at https://github.com/Fantasy-Studio/Paint-by-Example.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4f1502111d35aa6651dfaedfeb1184b3c3dd2fcb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, neutral): “Generative inpainting systems such as LaMa [54], Latent Diffusion inpainting [42], RePaint [33], Stable Diffusion Inpainting variants 2 , and FLUX.1 Fill 3 make such context-aware edits instantaneous; see also Palette [45] and Paint-by-Example [57].”
AnyDoor: Zero-shot Object-level Image Customization (2023, positive): “…Object Scene + Location Supervision [49] Single Image 15,572 ✗ High HFlickr [14] Single Image 4,833 ✗ High LVIS [21] Single Image 118,287 ✗ High SAM (subset) [26] Single Image 100,864 ✗ High x is the ground-truth image latent, t is the diffusion timestep, α t , σ t are denoising hyperparameters.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “We also compare different features quantitatively on an 800-pair subset of [69] benchmark.”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “…diffusion models have demonstrated superior performance across a range of generation tasks in various modalities, including text-to-image generation [40, 43, 41, 17], image manipulation [26, 34, 4, 56], video synthesis [18, 47, 22], text generation [28, 16, 59], 3D avatar synthesis [39, 53], etc.”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, methodology): “Furthermore, it can be integrated with models like LaMa [56] and PaintByExample [68] to achieve precise image erasure and customized image editing.”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, methodology): “ObjectStitch[42] and Paint-by-Example[53] leverage the CLIP[31] and propose diffusion-based image editing methods given image condition.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “Paint by example [144] proposes a semantic image composition problem reference image is semantically transformed and harmonized before blending into another image [144].”