Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
By Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Wei Wu and 3 more. Neural Information Processing Systems, 2023
316 citations recorded by the source corpus.
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can be easily customized for new concepts using low-rank adaptations (LoRAs). However, the utilization of multiple concept LoRAs to jointly support multiple customized concepts presents a challenge. We refer to this scenario as decentralized multi-concept customization, which involves single-client concept tuning and center-node concept fusion. In this paper, we propose a new framework called Mix-of-Show that addresses the challenges of decentralized multi-concept customization, including concept conflicts resulting from existing single-client LoRA tuning and identity loss during model fusion. Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, we introduce regionally controllable sampling, which extends spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address attribute binding and missing object problems in multi-concept sampling. Extensive experiments demonstrate that Mix-of-Show is capable of composing multiple customized concepts with high fidelity, including characters, objects, and scenes.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/5728ecb3a11c1586c4ae53e11ab395a0263eb5f4
Citations in the corpus
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “Our method can be easily integrated with personalized T2I models ( e.g ., DreamBooth [40], Mix-of-Show [11], etc.) which take 3-5 images as input and return a personalized T2I model.”
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, unknown): “While LoRA was initially introduced for adapting language models to down-stream tasks, recent efforts have illustrated its effectiveness when applied in conjunction with diffusion models [13, 41].”
- AnyDoor: Zero-shot Object-level Image Customization (2023, unknown): “Besides, since AnyDoor owns the high controllability for editing the specific local regions of the scene image, it is easy to be extended to multi-subject composition (middle row), which is a hot and challenging topic explored by many customized generation methods [2, 18, 28, 34].”
- PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, unknown): “Recent studies [19, 39, 40] also showed how to make different concepts appear in the generated image by training multiple LoRAs, which is different from our approach of semantically integrating multiple IDs.”