TokenFlow: Consistent Diffusion Features for Consistent Video Editing

By Michal Geyer, Omer Bar-Tal, Shai Bagon, Tali Dekel. International Conference on Learning Representations, 2023

490 citations recorded by the source corpus.

The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos. Webpage: https://diffusion-tokenflow.github.io/

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4761f173965195798cd3046ef4af608a83504e4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Andrew Zisserman
  5. Kristen Grauman
  6. Xiaogang Wang
  7. Trevor Darrell
  8. Jitendra Malik

Citations in the corpus