Pix2Video: Video Editing using Image Diffusion

By Duygu Ceylan, Chun-Hao Paul Huang, Niloy J. Mitra. IEEE International Conference on Computer Vision, 2023

390 citations recorded by the source corpus.

Image diffusion models, trained on massive image collections, have emerged as the most versatile image generator model in terms of quality and diversity. They support inverting real images and conditional (e.g., text) generation, making them attractive for high-quality image editing applications. We investigate how to use such pre-trained image models for text-guided video editing. The critical challenge is to achieve the target edits while still preserving the content of the source video. Our method works in two simple steps: first, we use a pre-trained structure-guided (e.g., depth) image diffusion model to perform text-guided edits on an anchor frame; then, in the key step, we progressively propagate the changes to the future frames via self-attention feature injection to adapt the core denoising step of the diffusion model. We then consolidate the changes by adjusting the latent code for the frame before continuing the process. Our approach is training-free and generalizes to a wide range of edits. We demonstrate the effectiveness of the approach by extensive experimentation and compare it against four different prior and parallel efforts (on ArXiv). We demonstrate that realistic text-guided video edits are possible, without any compute-intensive preprocessing or video-specific finetuning. https://duyguceylan.github.io/pix2video.github.io/.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/32a3c2fbd3e733bd0eea938517fec2ff8dc7c701

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “Existing and concurrent video editing methods that are based on image diffusion models have demonstrated that global appearance coherency across the edited frames can be achieved by extending the self-attention module to include multiple frames (e.g., [53, 19, 5, 34]).”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, neutral): “…(T2V) is a fundamental task in video generation, and there are other related video generation tasks such as video-driven ( i.e ., video editing) [11, 12, 18, 29, 43, 58, 62, 67, 69, 76, 80, 82, 101, 116, 119, 121, 127–129], image-driven ( i.e ., image-to-video) [14, 16, 23, 27, 32, 76, 77, 88,…”
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, methodology): “Similar to other zero-shot video editing methods [3, 20], we replace self-attention layers in the U-Net with crossframe attention layers to regularize the global style of I ′ i to match that of I ′ 1 and I ′ i−1.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, background): “These include text-to-video [56, 65, 81], image-to-video [78], video-to-video stylization [11, 15, 66], and video editing [10, 26, 68] among other video applications.”
Video-P2P: Video Editing with Cross-Attention Control (2023, neutral): “Besides, there have been some concurrent works and fol-lowers after our work, such as [5, 9, 21, 30, 45, 49, 52].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Based on these models, advanced image and video editing have been achieved through finetuning the model [41, 68, 5, 23, 61, 29] or controlling the inference process [30, 13, 34, 10, 35, 7, 31, 3].”