Text2LIVE: Text-Driven Layered Image and Video Editing

By Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, Tali Dekel. 2022

402 citations recorded by the source corpus.

We present a method for zero-shot, text-driven appearance manipulation in natural images and videos. Given an input image or video and a target text prompt, our goal is to edit the appearance of existing objects (e.g., object's texture) or augment the scene with visual effects (e.g., smoke, fire) in a semantically meaningful manner. We train a generator using an internal dataset of training examples, extracted from a single input (image or video and target text prompt), while leveraging an external pre-trained CLIP model to establish our losses. Rather than directly generating the edited output, our key idea is to generate an edit layer (color+opacity) that is composited over the original input. This allows us to constrain the generation process and maintain high fidelity to the original input via novel text-driven losses that are applied directly to the edit layer. Our method neither relies on a pre-trained generator nor requires user-provided edit masks. We demonstrate localized, semantic edits on high-resolution natural images and videos across a variety of objects and scenes.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Chen Change Loy
  7. Jitendra Malik
  8. Andrea Vedaldi

In Multimodal Learning

  1. Alexei A. Efros
  2. Aleksander Holynski
  3. Tim Brooks
  4. Daniel Cohen-Or
  5. Tali Dekel
  6. Amit H. Bermano
  7. Rinon Gal
  8. Ron Mokady

In Natural Language Generation

  1. Huan Sun
  2. Ron Mokady
  3. Amir Hertz

Citations in the corpus