Emu Edit: Precise Image Editing via Recognition and Generation Tasks

By Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, Yaniv Taigman. Computer Vision and Pattern Recognition, 2023

338 citations recorded by the source corpus.

Instruction-based image editing holds immense potential for a variety of applications, as it enables users to perform any editing operation using a natural language instruction. However, current models in this domain often struggle with accurately executing user instructions. We present Emu Edit, a multitask image editing model which sets state-of-the-art results in instruction-based image editing. To develop Emu Edit we train it to multitask across an unprecedented range of tasks, such as region-based editing, free-form editing, and Computer Vision tasks, all of which are formulated as generative tasks. Additionally, to enhance Emu Edit's multitask learning abilities, we provide it with learned task embeddings which guide the generation process towards the correct edit type. Both these elements are essential for Emu Edit's outstanding performance. Furthermore, we show that Emu Edit can generalize to new tasks, such as image inpainting, super-resolution, and compositions of editing tasks, with just a few labeled examples. This capability offers a significant advantage in scenarios where high-quality samples are scarce. Lastly, to facilitate a more rigorous and informed assessment of instructable image editing models, we release a new challenging and versatile benchmark that includes seven different image editing tasks. 1

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5bcb0153dd0840113eb27d4d6f753414ef656a03

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, methodology): “We finetune the 2B parameter base model on a dataset consisting of image-to-image editing tasks similar to the distribution of the InstructPix2Pix dataset (Brooks et al., 2023) as well as inpainting, segmentation, colorization, deblurring and controlnet tasks similar to Emu Edit and Palette (Sheynin”
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, methodology): “Manual examination of random examples from the EmuEdit test set [Sheynin et al., 2024], shown Figure 6 and Appendix 4.5, reveals that our fine-tuned Transfusion model performs image edits as instructed.”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, neutral): “Other benchmarks like Emu-Edit [51] use lower-resolution images with unrealistic distributions and focus solely on editing tasks, while DreamBench [39] lacks broad coverage and GEdit-bench [31] does not represent the full scope of modern multimodal models.”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, positive): “Similar to [52] we use additional data from bidirectional controlnet tasks (canny edges, keypoints, semantic segmentation, depth maps, HED lines) as well as object segmentation.”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, background): “While gathering this data incurs negligible cost for small datasets like CIFAR-10, it becomes a significant bottleneck with large-scale text-to-image synthesis tasks, or models with complex conditioning [54–56].”