DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

By Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman. 2022

4,398 citations recorded by the source corpus.

Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for “personalization” of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

  1. Eli Shechtman
  2. Richard Zhang
  3. Jun-Yan Zhu
  4. Daniel Cohen-Or
  5. Alexei A. Efros
  6. Ying Shan
  7. Xintao Wang
  8. Ziwei Liu

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Jitendra Malik
  7. Chen Change Loy
  8. Andrea Vedaldi

In Generative Models

  1. Eli Shechtman
  2. Jun-Yan Zhu
  3. Richard Zhang
  4. Daniel Cohen-Or
  5. Arash Vahdat
  6. Ming-Hsuan Yang
  7. Alexei A. Efros
  8. Karsten Kreis

In Multimodal Learning

  1. Yu Qiao
  2. Ying Shan
  3. Steven C. H. Hoi
  4. Hongsheng Li
  5. Junnan Li
  6. Peng Gao
  7. Dongxu Li
  8. Jason Baldridge

Citations in the corpus