Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting

By Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut and 4 more. Computer Vision and Pattern Recognition, 2022

295 citations recorded by the source corpus.

Text-guided image editing can have a transformative impact in supporting creative applications. A key challenge is to generate edits that are faithful to input text prompts, while consistent with input images. We present Imagen Editor, a cascaded diffusion model built, by fine-tuning Imagen [36] on text-guided image inpainting. Imagen Editor's edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. In addition, Imagen Editor captures fine details in the input image by conditioning the cascaded pipeline on the original high resolution image. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting. EditBench evaluates inpainting edits on natural and generated images exploring objects, attributes, and scenes. Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment – such that Imagen Editor is preferred over DALL-E 2 [31] and Stable Diffusion [33] – and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f1c39410893794ee3643efa85be1816964aa85ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “For text-to-image (T2I) models, several benchmarks [6, 7, 44, 61, 86, 99] are proposed to assess various capabilities like compositionality [44] and editing ability [7, 99].”
Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, background): “Imagen Editor [23] and SmartBrush [24] finetune the text-to-image model to be conditioned on both the input image and mask.”
Step1X-Edit: A Practical Framework for General Image Editing (2025, unknown): “EditBench [57] 240 [40] 6,538 Table 1: Key Attributes of Open-source Edit Benchmarks .”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, unknown): “We now test VQAScore on five text-to-image evaluation benchmarks (TIFA160 [25], Pick-a-Pic [33], and DrawBench [65], EditBench [78], COCO-T2I [43]) to measure its correlation (or agreement) with human judgments of alignment.”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Diffusion models have been successfully applied to image generation [25–27], speech generation [8,35], and video generation [24,58,68], and have found applications in various image processing areas, including image colorization, super-resolution, inpainting, and semantic editing [49,51,61,69].”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, background): “…al., 2022), scene graphs (Yang et al., 2022), sketches (Voynov et al., 2022), depthmaps (stability.ai, 2022), and inpainting masks (Xie et al., 2022; Wang et al., 2022a), or by ﬁnetuning the pretrained models on a few subject-speciﬁc data (Gal et al., 2022; Mokady et al., 2022; Ruiz et al., 2022).”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, background): “Imagen Editor [37] trains diffusion editing models by inpainting the masked objects.”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “ROSIE [176] use the Imagen editor [177] to modify training images to add additional distractors and unseen objects and backgrounds to train robust imitation learning policies.”