DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
By Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman. 2022
4,398 citations recorded by the source corpus.
Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for “personalization” of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Eli Shechtman
- Richard Zhang
- Jun-Yan Zhu
- Daniel Cohen-Or
- Alexei A. Efros
- Ying Shan
- Xintao Wang
- Ziwei Liu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Chen Change Loy
- Andrea Vedaldi
In Generative Models
- Eli Shechtman
- Jun-Yan Zhu
- Richard Zhang
- Daniel Cohen-Or
- Arash Vahdat
- Ming-Hsuan Yang
- Alexei A. Efros
- Karsten Kreis
In Multimodal Learning
Citations in the corpus
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Learning to follow instructions Our method differs from existing text-based image editing works [6,13,17,27,38,52] in that it enables editing from instructions that tell the model what action to perform, as opposed to text labels, captions or descriptions of input/output images.”
- SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Notably, generative models using multi-modal input have emerged as a remarkable innovation [5], [6], [7], [8], [9], [10].”
- Aligning Text-to-Image Models using Human Feedback (2023, background): “Fine-tuning with few images (Ruiz et al., 2022; Kumari et al., 2022; Gal et al., 2022) for personalization of text-to-image diffusion models is also related with our work.”
- Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, methodology): “Previous works, such as Dreambooth [10] and Custom Diffusion [11], have primarily focused on object customization.”
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, positive): “Direct tuning enables more flexible adaptation, not only for few-shot customization scenarios like DreamBooth [39] but also for object detection, as in 3DiffTection [53].”
- Inversion-based Style Transfer with Diffusion Models (2022, background): “[39] implant a subject into the output domain of the textto-image diffusion model so that it can be synthesized in”
- Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “These models enable various image manipulation tasks [3,22,30,52] previously unassailable, allowing image editing for general images with the guidance of text prompt.”
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators (2023, methodology): “After that, to show the universality of our method, we use it in combination with ControlNet [47] and Dream-Booth [31] diffusion models for generating conditional and specialized videos.”
- Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “In order to synthesize novel scenes with certain concepts or subjects, [68], [69] introduces several reference images with the desired concepts, then inverts the reference images to the textual descriptions.”
- Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Based on these models, advanced image and video editing have been achieved through finetuning the model [41, 67, 5, 23, 61, 29] or controlling the inference process [30, 13, 34, 10, 35, 7, 31, 3].”
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and human-centric motion generation [39, 6, 32, 61, 5].”
- Personalize Segment Anything Model with One Shot (2023, methodology): “We follow most model hyperparameters and training configurations in DreamBooth [45], including a 10−6 learning rate, a batch size 1, and a 200-image regularization dataset.”
- DIRE for Diffusion-Generated Image Detection (2023, neutral): “Besides unconditional image generation, there are also plenty of text-to-image generation works based on diffusion models [37, 14, 35, 39, 5, 38].”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “Diffusion models has various applications [62], [199], [32], [38], [155], while the most notable is stable diffusion [153], which bridges the gap between text and image with its robust capability to generate coherent and contextually relevant images directly from textual descriptions.”
- CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “AnimateDiff (Guo et al., 2023b) utilizes a pluggable motion module to enable high-quality animation creation on personalized image backbones (Ruiz et al., 2023).”
- DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, neutral): “…(Jing et al., 2019), inserting objects in images (Gafni & Wolf, 2020; Brown et al., 2022), image-to-image translation (Zhu et al., 2017; Sa-haria et al., 2022a), inpainting (Yu et al., 2018), scene graph manipulation (Dhamo et al., 2020), and placing subjects in novel contexts (Ruiz et al., 2022).”
- AnyDoor: Zero-shot Object-level Image Customization (2023, neutral): “We also compare Stable Diffusion [41], which is a text-to-image model, and we use its inpainting version and give detailed text descriptions as the condition to conduct the generation for the text-described target.”