An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
By Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. 2022
2,832 citations recorded by the source corpus.
Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new"words"in the embedding space of a frozen text-to-image model. These"words"can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io
Subfields: Computer Vision and Pattern Recognition, Literature and Literary Theory, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Literature and Literary Theory
- Tali Dekel
- Daniel Cohen-Or
- Yu Qiao
- Andrea Vedaldi
- Graham Neubig
- Alexei A. Efros
- Yuval Alaluf
- Shixiang Shane Gu
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Lior Wolf
- Eliya Nachmani
- Sergey Levine
- Ruslan Salakhutdinov
- Jianfeng Gao
- Adam Polyak
In Multimodal Learning
Citations in the corpus
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Future work may also involve meta-learning the multimodal exchanges themselves, expanding the intermediate representations from discrete to continuous (Gal et al., 2022), or extending them to include additional modalities beyond text, e.”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “More recently, personalization (Gal et al., 2022; Benhamdi et al., 2017) efforts can be found in vision and graphics, which allows to inject unique objects into new scenes, transform them across different styles, and even produce new products.”
- InstantID: Zero-shot Identity-Preserving Generation in Seconds (2024, methodology): “Previous subject-driven approaches like DreamBooth [18], Textual Inversion [4], ELITE [22], E4T [5], and ProFusion [28] fine-tune a special prompt token S ∗ to describe the target concepts during the fine-tuning process.”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “Textual Inversion [49] addresses this by finding a new pseudo-word S ˚ to soft prompt discussed in Section III-A2) that represent new, specific concepts in the textual embedding space of pre-trained text-to-image diffusion models.”
- GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “This calibrates the model to the victim artist’s style, identifying important features related to the victim style and associating these regions in the feature space with the victim artist’s name [29, 72].”
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, positive): “Align ↑ Face Sim ↑ Recon ↑ Time (s) ↓ TI [11] 0.2556 0.1130 0.7832 ∼ 1500 DB [30] 0.3088 0.2408 0.8335 ∼ 600 ELITE [38] 1.”
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, positive): “Recent works [3, 7, 12, 21, 24,33,45] explore pertrained T2I diffusion models for textdriven image editing, such as object or background replacement, style or color transfers, attribute modifications, etc.”
- Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Textual Inversion [17] and Dream-Booth [51] synthesize novel views of a given subject given 3 – 5 images of the subject and a target text (rather than edit a single image), with DreamBooth requiring additional generated images for fine-tuning the models.”
- EDICT: Exact Diffusion Inversion via Coupled Transformations (2022, methodology): “Another set of methods [6, 11, 25] finetune the model with the target image and/or learn a new conditioning prompt to enable indirect image editing via sampling.”
- StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, background): “Methods for “personalizing” diffusion models have become popular (Ruiz et al., 2022; Gal et al., 2022).”
- Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “Specifically, [68] inverts the shared concept in a couple of reference images into the text (embedding) space, i.”
- Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, methodology): “Textual-Inversion As our Latent-NeRF is supervised by Stable-Diffusion, we can also use Textual Inversion [16] to-kens as part of the input text prompt.”
- Inversion-based Style Transfer with Diffusion Models (2022, methodology): “Ours TexIn [15] SDM [42] MLP w/o drop Accuracy(%)↑ 78.”
- Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “The research community has also taken steps to control text-to-image models with spatial masks [6, 20], image editing instructions [10], personalization via finetuning [21, 74], etc .”
- Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (2023, background): “We follow the setting investigated with soft prompts in Gal et al. (2022) but with our hard prompts.”
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (2022, background): “DreamBooth [39] and Textual Inversion [14] personalize a pre-trained text-to-image diffusion model given a few userprovided images.”
- Effective Data Augmentation With Diffusion Models (2023, positive): “only a handful of labelled examples are observed Gal et al. (2022). Rather than generate synthetic images from scratch, we use image-to-image transformations that splice real images into the reverse diffusion process following prior work in SDEdit Meng et al. (2022a). Given a reverse diffusion proce”