InstructPix2Pix: Learning to Follow Image Editing Instructions

By Tim Brooks, Aleksander Holynski, Alexei A. Efros. 2022

3,195 citations recorded by the source corpus.

We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models—a language model (GPT-3) and a text-to-image model (Stable Diffusion)—to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per-example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Visual Instruction Tuning (2023, methodology): “In the image editing domain, given an input image and a written instruction that tells the agent what to do, InstructPix2Pix [6] edits images by following the human instructions.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “To fuse the vision processing ability into such LLMs, several challenges are lying ahead since it is hard to train either large language models or vision models, and the well-designed instructions [4, 55, 21] and cumbersome conversions [30, 52] are required to connect different modalities.”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “We instantiate our approach with Instruct Pix2Pix [9], though other image-editing models could also be used.”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “In addition, these methods lack the ability to mimic the appearance of subjects in a given reference set.”
Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, methodology): “Some recent studies [26, 30, 3] use the CLIP [29] model as a proxy for human judgment to evaluate the alignment between generated images and text prompts.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “Based on a large model (GPT-3) and a text-to-image model (Stable diffusion), [146] first generates a dataset for this new task, and trains a conditional diffusion model InstructPix2Pix which generalize to real images well.”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, methodology): “However, such edit turn could be easily obtained automatically [4] due to its less photorealism and more artistic nature.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, neutral): “Existing work produces 3D models that are either trained per category [5, 15, 54, 66, 72], or are limited to single image inputs containing an object [29, 30, 48], preventing their use as a general prior for 3D scene reconstruction.”
Muse: Text-To-Image Generation via Masked Generative Transformers (2023, background): “A number of techniques have been developed on diffusion models to enable editing, personalization and inversion to token space (Gal et al., 2022a; Meng et al., 2021; Ruiz et al., 2022; Kawar et al., 2022; Brooks et al., 2022; Hertz et al., 2022; Mokady et al., 2022).”
Generative Multimodal Models are In-Context Learners (2023, methodology): “Additionally, we leverage InstructPix2Pix constructed by [10] for image editing tasks.”
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, methodology): “Thus, these methods are naturally compatible with pre-trained diffusion variants like InstructPix2Pix [2] or ControlNet to accept more flexible conditions like depth and edges.”
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models (2023, background): “–18] have emerged as a popular choice for synthesizing high-quality images based on textual inputs.”
Pix2Video: Video Editing using Image Diffusion (2023, background): “, image embedding conditioning [6]) and a smart anchor update mechanism is a potential direction.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “4 Personalization,Customization, and Control of Pretrained Diffusion Model Because state-of-the-art image diffusion models are dominated by text-to-image methods, the most straight-forward ways to enhance the control over a diffusion model are often text-guided [38, 24, 2, 3, 23, 43, 16].”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, neutral): “GaussianEditor-DDS, due to the more challenging control of guidance offered by DDS loss [14] compared to Instruct-pix2pix [4], tends to result in oversaturated colors and less precise editing outcomes.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Using compositional classifier-free guidance [8, 19, 32] to amplify the individual impact of action conditions may also help (at a cost of increased inference compute).”
OmniGen: Unified Image Generation (2024, positive): “Instruct-Pix2Pix [3], InstructSD [48] and EMU-edit [58] explore performing general image editing tasks through instructions.”