An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

By Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. 2022

2,832 citations recorded by the source corpus.

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new"words"in the embedding space of a frozen text-to-image model. These"words"can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io

Subfields: Computer Vision and Pattern Recognition, Literature and Literary Theory, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Jitendra Malik
  7. Andrea Vedaldi
  8. Andrew Zisserman

In Literature and Literary Theory

  1. Tali Dekel
  2. Daniel Cohen-Or
  3. Yu Qiao
  4. Andrea Vedaldi
  5. Graham Neubig
  6. Alexei A. Efros
  7. Yuval Alaluf
  8. Shixiang Shane Gu

In Natural Language Processing

  1. Graham Neubig
  2. Mohit Bansal
  3. Lior Wolf
  4. Eliya Nachmani
  5. Sergey Levine
  6. Ruslan Salakhutdinov
  7. Jianfeng Gao
  8. Adam Polyak

In Multimodal Learning

  1. Ying Shan
  2. Yu Qiao
  3. Steven C. H. Hoi
  4. Hongsheng Li
  5. Junnan Li
  6. Peng Gao
  7. Xintao Wang
  8. Dongxu Li

Citations in the corpus