BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

By Dongxu Li, Junnan Li, Steven C. H. Hoi. Neural Information Processing Systems, 2023

559 citations recorded by the source corpus.

Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Code and models will be released at https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion. Project page at https://dxli94.github.io/BLIP-Diffusion-website/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dc0c132b273456b288a785414db2fa72edf87b1a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OmniGen: Unified Image Generation (2024, unknown): “The Chain-of-Thought (CoT) method can significantly boost the performance of LLMs by decomposing the task into multiple steps and sequentially solving each step to obtain an accurate final answer.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “…and potential of D REAM LLM in generating consistent images that preserve subject features, we fine-tune stage II pretrained D REAM LLM on subject representation learning data constructed by following the recently proposed controllable image generation method BLIP-Diffusion (Li et al., 2023c).”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “The second is encoder-based methods [6, 15, 19, 22, 38, 39, 41], which learn image embeddings from the input images as the representation of the concept.”
Generative Multimodal Models are In-Context Learners (2023, background): “With the remarkable progress in open-sourced LLMs, [42, 43, 49, 94] show promising results by connecting vision encoders and LLMs with a small intermediate model.”
AnyDoor: Zero-shot Object-level Image Customization (2023, unknown): “Recently, BLIP-Diffusion [29] leverages BLIP-2 [30] to align image and text, thus supporting using zero-shot subject-driven generation.”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “For systematic benchmarking, recent work [3, 4, 39, 64, 68, 80] adopts metrics such as CLIPScore [21, 61], which measures the (cosine) similarity of the embedded image and text prompt.”
OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, unknown): “To address this limitation, some works have explored image-based control methods [2, 14, 15, 18, 26– 30, 37, 42, 46, 50, 52, 54, 56, 57, 57, 60, 61, 63], enabling users to specify their intentions through reference images or visual hints, offering more precise guidance than text alone [44].”