InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning

By Jing Shi, Wei Xiong, Zhe L. Lin, Hyun Joon Jung. Computer Vision and Pattern Recognition, 2023

421 citations recorded by the source corpus.

Recent advances in personalized image generation have enabled pre-trained text-to-image models to learn new concepts from specific image sets. However, these methods often necessitate extensive test-time finetuning for each new concept, leading to inefficiencies in both time and scalability. To address this challenge, we introduce Instant-Booth, an innovative approach leveraging existing text-to-image models for instantaneous text-guided image personalization, eliminating the need for test-time finetuning. This efficiency is achieved through two primary innovations. Firstly, we utilize an image encoder that transforms input images into a global embedding to grasp the general concept. Secondly, we integrate new adapter layers into the pre-trained model, enhancing its ability to capture intricate identity details while maintaining language coherence. Significantly, our model is trained exclusively on textimage pairs, without reliance on concept-specific paired images. When benchmarked against existing finetuning-based personalization techniques like DreamBooth and TextualInversion, InstantBooth not only shows comparable proficiency in aligning language with image, maintaining image quality, and preserving the identity but also boasts a 100-fold increase in generation speed. Project Page: https://jshi31.github.io/InstantBooth/

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9733025aea2ba71792be10c18d635e8fc1455e31

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

AnyDoor: Zero-shot Object-level Image Customization (2023, neutral): “Thus we generate 2,400 images for the object-scene combinations.”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, methodology): “While these methods are mainly based on parameter tuning, several works have also tried to learn a more general encoder for concept personalization [10, 14, 27].”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Other works have also investigated training-free approaches for fast adaptation [17, 73, 11, 27, 55].”
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing (2023, methodology): “Concurrent to our effort, the work [11, 17, 18] pre-train the diffusion model on domain-specific images, such as cat and human face images.”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, background): “Another approach to concept customization, as explored in studies by Instantbooth [14], ELITE [15], and Jia et al. [16], focuses on achieving fast test-time customization.”
OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, neutral): “…show clear task bias – they typically work well for either spatially aligned controls (e.g., edge-guided or depth-guided generation [2, 15, 29, 30, 37, 42, 52, 61, 63]) or spatially un-aligned ones (e.g., style transfer or subject-driven generation [18, 27, 28, 50, 60][14, 33]), but rarely both.”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, methodology): “These methods either utilize personalization datasets [10, 63] for training or encode the images to be customized in the semantic space [9, 29, 43, 62, 70, 72].”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Similar to adding controls in T2I [48, 61, 88, 93], introducing control signals in T2V, such as structure [17, 83], pose [44, 94], and Canny edge [39], has been increasingly receiving much attention.”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “Inspired by existing visual conditioning works [44, 55], we utilize the full patch visual to-kens F vis = { f i } K i =0 from the last layer of the CLIP image ViT [17], which are believed to encompass much richer information about the image.”