IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

By Hu Ye, Jun Zhang, Siyi Liu, Wei Yang. arXiv.org, 2023

1,685 citations recorded by the source corpus.

Recent years have witnessed the strong power of large text-to-image diffusion models for the impressive generative capability to create high-fidelity images. However, it is very tricky to generate desired images using only text prompt as it often involves complex prompt engineering. An alternative to text prompt is image prompt, as the saying goes:"an image is worth a thousand words". Although existing methods of direct fine-tuning from pretrained models are effective, they require large computing resources and are not compatible with other base models, text prompt, and structural controls. In this paper, we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Despite the simplicity of our method, an IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model. As we freeze the pretrained diffusion model, the proposed IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. With the benefit of the decoupled cross-attention strategy, the image prompt can also work well with the text prompt to achieve multimodal image generation. The project page is available at \url{https://ip-adapter.github.io}.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2854e5bab8e6f36e54c64456628a9559bf67019e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, neutral): “To evaluate the enhancement brought by the proposed appearance encoding strategy, we replace the appearance encoder in MagicAnimate with CLIP [21] and IP-Adapter [44] to establish baselines.”
AnyDoor: Zero-shot Object-level Image Customization (2023, positive): “Some concurrent works [30, 58, 61] also explore using one reference image to customize the generation results but fail to keep the fine details.”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, positive): “The remarkable performance of these models can be attributable to high-quality large-scale text-image datasets [7, 60, 61], the continuous upgrades of foundational models [8, 48], conditioning encoders [27, 50, 51], and the improvement of controllability [36, 45, 73, 75].”
InstantID: Zero-shot Identity-Preserving Generation in Seconds (2024, methodology): “IP-Adapter-FaceID IP-Adapter-FaceID [24] uses face ID embedding from a face recognition model instead of CLIP image embedding to retain ID consistency.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “Recognizing that MVDream is primarily trained on text embeddings, we introduced a multi-layer perceptron (MLP) θ g , functioning as an adaptor similar to IP-Adaptor [51], following the CLIP image global embedding.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “It then projects the latent code and text embeddings into Query, Key, and Value matrices: Here, W q ∈ R f × d , W k , W v ∈ R d × d represent the weight matrices of the three linear layers, and d (cid:48) is the dimension of Query, Key, and Value embeddings.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Similar to adding controls in T2I [48, 61, 88, 93], introducing control signals in T2V, such as structure [17, 83], pose [44, 94], and Canny edge [39], has been increasingly receiving much attention.”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, neutral): “Early works on such training-free subject-driven image synthesis include IP-Adapter [58] or retrieval-augmented diffusion variants [7, 3].”
OmniGen: Unified Image Generation (2024, neutral): “In this section, we present the details of OmniGen framework, including the model architecture and training method.”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “FlexiTex [39] and EASI-Tex [42] both utilize an IP-Adapter [108] for image prompt injection.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “IP-Adapter [202] identifies limitations in current approaches (e.g., ControlNet and T2I-Adapter) which project condition signals into the cross-attention modules.”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, methodology): “We also conduct a baseline that combines Stable Diffusion, ControlNet, IP-Adapter[56] and Animate-Diff, named SD-I2V.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “Typically, these control signals are first fed into an additional encoder Φ s and then injected into the generator through various operations (Zhang et al., 2023a; Mou et al., 2023; Ye et al., 2023).”
OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, neutral): “To address this limitation, some works have explored image-based control methods [2, 14, 15, 18, 26– 30, 37, 42, 46, 50, 52, 54, 56, 57, 57, 60, 61, 63], enabling users to specify their intentions through reference images or visual hints, offering more precise guidance than text alone [44].”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, background): “Aside from consuming vast computational resources, these models are difficult to integrate with the burgeoning community models and downstream tools [59,61].”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “The image embedding is extracted using CLIP [14] and injected into the SD UNet through cross attention [54], similar to the injection of text embeddings.”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, background): “Moreover, in the endeavor to enhance control over visual generation, recent studies such as ControlNet [53], T2I-Adapter [25], and IP-Adapter [49] have delved into the incorporation of supplementary encoder layers.”
CLeaR: A Unified Framework for Resolving the Leakage-Degradation Dilemma in Style Transfer (2026, unknown): “Standard condition injectors such as IP-Adapter [47] are trained on ordinary image-text pairs rather than on content-reduced style anchors, so the denoising trajectory may drift from the intended style or reintroduce leaked content.”