SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

By Ligong Han, Yinxiao Li, Peyman Milanfar, Dimitris N. Metaxas, Feng Yang. IEEE International Conference on Computer Vision, 2023

407 citations recorded by the source corpus.

Diffusion models have achieved remarkable success in text-to-image generation, enabling the creation of high-quality images from text prompts or other modalities. However, existing methods for customizing these models are limited by handling multiple personalized subjects and the risk of overfitting. Moreover, their large number of parameters is inefficient for model storage. In this paper, we propose a novel approach to address these limitations in existing text-to-image diffusion models for personalization. Our method involves fine-tuning the singular values of the weight matrices, leading to a compact and efficient parameter space that reduces the risk of overfitting and language-drifting. We also propose a Cut-Mix-Unmix data-augmentation technique to enhance the quality of multi-subject image generation and a simple text-based image editing framework. Our proposed SVDiff method has a significantly smaller model size compared to existing methods (≈2,200 times fewer parameters compared with vanilla DreamBooth), making it more practical for real-world applications.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6ae34677bc41e1818a899583b25e379dadc42a85

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “We encode X into a global concept embedding c g via Concept Encoder E g and obtain”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, methodology): “Following this, SVDiff [12] introduces data augmentation to prevent concept mixing in co-training multi-concepts, and Cones [13] discovers concept neurons that can be added to support multiple concepts.”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, positive): “Given that both pioneer works require substantial time for fine-tuning, some studies have attempted to expedite the process of personalized customization by reducing the number of parameters needed for tuning [20, 34, 57, 74] or by pre-training with large datasets [17, 56, 65].”