Multi-Concept Customization of Text-to-Image Diffusion

By Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu. Computer Vision and Pattern Recognition, 2022

1,364 citations recorded by the source corpus.

While generative models produce high-quality images of concepts learned from a large-scale database, a user often wishes to synthesize instantiations of their own concepts (for example, their family, pets, or items). Can we teach a model to quickly acquire a new concept, given a few examples? Furthermore, can we compose multiple new concepts together? We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts while enabling fast tuning (~ 6 minutes). Additionally, we can jointly train for multiple concepts or combine multiple fine-tuned models into one via closed-form constrained optimization. Our fine-tuned model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings. Our method outperforms or performs on par with several baselines and concurrent works in both qualitative and quantitative evaluations, while being memory and computationally efficient.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/144eca44e250cc462f6fc3a172abb865978f66f5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation (2023, neutral): “Albeit flexible generation has been achieved, existing methods still require several or tens of minutes to learn a single concept.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, neutral): “The first is test-time finetuning-based methods [5, 9, 11–13, 17, 30, 31, 36, 37].”
Erasing Concepts from Diffusion Models (2023, background): “Recently, it has been shown that text-conditional diffusion models can be edited by associating a token for a new subject trained using only a handful of images [13, 20, 35].”
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing (2023, background): “One known inefficiency of this approach is that it requires reiterating hundreds [9, 10] or thousands [8] tedious fine-tuning steps for each new subject, which hinders it from efficiently scaling to a wide range of subjects.”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, methodology): “Custom Diffusion [11] proposes co-training of multiple concepts or constrained optimization of several existing concept models.”
Video-P2P: Video Editing with Cross-Attention Control (2023, neutral): “Textual Inversion [11], Dream-Booth [35], and Custom-Diffusion [20] learn special to-kens for personalized concepts and generate related images.”
Aligning Text-to-Image Models using Human Feedback (2023, background): “Fine-tuning with few images (Ruiz et al., 2022; Kumari et al., 2022; Gal et al., 2022) for personalization of text-to-image diffusion models is also related with our work.”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, neutral): “Given that both pioneer works require substantial time for fine-tuning, some studies have attempted to expedite the process of personalized customization by reducing the number of parameters needed for tuning [20, 34, 57, 74] or by pre-training with large datasets [17, 56, 65].”
AnyDoor: Zero-shot Object-level Image Customization (2023, positive): “Customized synthesis methods [18, 27, 33, 34, 42] are able to conduct generations for the new concepts but could not be specified for a location of a given scene.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, background): “Another related line of work aims to introduce specific concepts to a pre-trained text-to-image model by learning to map a set of images to a “word” in the embedding space of the model [18, 25, 41].”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, methodology): “…training the StableDiffu-sion [51] model with multiple concepts simultaneously (randomly choosing one concept at each data sampling iteration), the model tends to mix their styles when rendering them in one image for difficult compositions or subjects of similar categories [32] (as shown in Fig.”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, methodology): “Custom Diffusion [16] improves ﬁne-tuning efﬁciency by updating only a small subset of parameters and allowing concept merging through closed-form optimization.”
Personalize Segment Anything Model with One Shot (2023, methodology): “The raw images are collect from the training data of subjectdriven diffusion models: DreamBooth [45], Textual Inversion [12], and Custom Diffusion [28].”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, background): “Recent text-to-visual systems [2– 4, 16, 17, 23, 31, 32, 34, 36–38, 53, 64, 65, 68, 80] mostly report CLIPScore [21], which measures (cosine) similarity of the embedded image and text prompt.”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Based on these models, advanced image and video editing have been achieved through finetuning the model [41, 68, 5, 23, 61, 29] or controlling the inference process [30, 13, 34, 10, 35, 7, 31, 3].”
Ablating Concepts in Text-to-Image Diffusion Models (2023, methodology): “Several recent works also explore fine-tuning text-to-image models to learn personalized or unseen concepts [33, 17, 55, 18] given a few exemplar images.”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, background): “Among these generative models, diffusion-based models are well open-sourced and popularly applied to several downstream tasks, such as image editing [8, 19] and customized generation [6, 16, 31, 37].”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “…et al. 2022b; Ho et al. 2022) have been popularly used in numerous downstream tasks, including personalized image generation (Ruiz et al. 2023; Kumari et al. 2023), image editing (Hertz et al. 2022; Brooks, Holynski, and Efros 2023; Kawar et al. 2023), image inpainting (Yang et al. 2022a) and…”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, neutral): “This property enables personalization of generative image and video models without the need for finetuning [43] or LoRA training [19, 27, 20].”