Text2LIVE: Text-Driven Layered Image and Video Editing
By Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, Tali Dekel. 2022
402 citations recorded by the source corpus.
We present a method for zero-shot, text-driven appearance manipulation in natural images and videos. Given an input image or video and a target text prompt, our goal is to edit the appearance of existing objects (e.g., object's texture) or augment the scene with visual effects (e.g., smoke, fire) in a semantically meaningful manner. We train a generator using an internal dataset of training examples, extracted from a single input (image or video and target text prompt), while leveraging an external pre-trained CLIP model to establish our losses. Rather than directly generating the edited output, our key idea is to generate an edit layer (color+opacity) that is composited over the original input. This allows us to constrain the generation process and maintain high fidelity to the original input via novel text-driven losses that are applied directly to the edit layer. Our method neither relies on a pre-trained generator nor requires user-provided edit masks. We demonstrate localized, semantic edits on high-resolution natural images and videos across a variety of objects and scenes.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Chen Change Loy
- Jitendra Malik
- Andrea Vedaldi
In Multimodal Learning
- Alexei A. Efros
- Aleksander Holynski
- Tim Brooks
- Daniel Cohen-Or
- Tali Dekel
- Amit H. Bermano
- Rinon Gal
- Ron Mokady
In Natural Language Generation
Citations in the corpus
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “[17] Ours We compare with recent image editing approaches SDEdit [38], Text2Live [6], and Prompt-to-Prompt [17].”
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “Fig 7 shows an additional qualitative comparison of our method to Text2LIVE [1] and to propagating an edit from a single keyframe (edited using [51]) to the rest of the video using [17].”
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “More following work are proposed using transformers [10, 11] or diffusion models [18, 3, 5, 17, 19, 23, 25] to generate high-fidelity images from text.”
- CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “CelebV-HQ provides the possibility of improving Video Generation/Editing, such as unconditional face generation [24,64,44,45,75,94,73], text-to-video generation [51,85,30,31,3], video facial attributes editing [13,34,70,76,88], face reenactment [87,71,95,83], and face swapping [49,101,22,60,89].”
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators (2023, methodology): “With the rise of text-guided image editing methods such as Prompt2Prompt [ 9], Instruct-Pix2Pix [2], SDEdit [20], etc., text-guided video editing approaches emerged [1, 16, 42].”
- Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Based on these models, advanced image and video editing have been achieved through finetuning the model [41, 67, 5, 23, 61, 29] or controlling the inference process [30, 13, 34, 10, 35, 7, 31, 3].”
- Diffusion Self-Guidance for Controllable Image Generation (2023, background): “These details are often supplied in the form of reference images along with a new textual prompt [2, 4] or other forms of conditioning [1, 31, 39].”
- DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, methodology): “Others (Crowson et al., 2022; Couairon et al., 2022; Patashnik et al., 2021; Bar-Tal et al., 2022) rely on optimization of the image itself, or a latent representation of it, to modify an image based on a high-level multimodal objective in an embedding space, typically using CLIP (Radford et al.”
- Pix2Video: Video Editing using Image Diffusion (2023, neutral): “Jamriska et al. [20] no no no Text2Live [3] 7-8 hours 30 min yes Tune-a-Video [58] 75sec 10 min no ours 125sec no no “a grizzly bear walking around the lake, foggy day” “a group of rainbow fish swimming in an aquarium” “a kite-surfer in the magical starry ocean with aurora borealis in the…”
- MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, methodology): “(3) Text2LIVE [2] performs well in L1 and L2 evaluations, likely due to the addition of an extra editing layer that minimizes changes to the source image.”
- Structure and Content-Guided Video Synthesis with Diffusion Models (2023, methodology): “Text2Live [5] allows editing input videos us-ing text prompts by decomposing a video into neural layers [21].”
- An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “, 2022), video manipulation (Tzaban et al., 2022; Bar-Tal et al., 2022), motion synthesis (Tevet et al.”
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “For video local editing, layer-atlas based methods [4,24] show a promising direction by editing the video on a flattened texture map.”
- Implicit Diffusion Models for Continuous Super-Resolution (2023, neutral): “In image super-resolution, regression-based methods, such as EDSR [23], RRDB [45], and SWinIR [21], directly learn a mapping from LR to HR images with an MSE loss.”
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, background): “, 2022), many diffusion-based image editing methods have been proposed (Hertz et al., 2022; Cao et al., 2023; Mao et al., 2023; Kawar et al., 2023; Parmar et al., 2023; Liew et al., 2022; Mou et al., 2023; Tumanyan et al., 2023; Brooks et al., 2023; Meng et al., 2021; Bar-Tal et al., 2022).”
- Video-P2P: Video Editing with Cross-Attention Control (2023, positive): “Text2Live [2] employs CLIP to edit textures in videos but struggles with significant semantic changes.”
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “While most works that require only text are limited to global editing [14, 31], Bar-Tal et al. [5] proposed a text-based localized editing technique without using masks, showing impressive results.”