FateZero: Fusing Attentions for Zero-shot Text-based Video Editing

By Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, Qifeng Chen. IEEE International Conference on Computer Vision, 2023

548 citations recorded by the source corpus.

The diffusion-based generative models have achieved remarkable success in text-based image generation. However, since it contains enormous randomness in generation progress, it is still challenging to apply such models for real-world visual content editing, especially in videos. In this paper, we propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. To edit videos consistently, we propose several techniques based on the pre-trained models. Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information. These maps are directly fused in the editing process rather than generated during denoising. To further minimize semantic leakage of the source video, we then fuse self-attentions with a blending mask obtained by cross-attention features from the source prompt. Furthermore, we have implemented a reform of the self-attention mechanism in denoising UNet by introducing spatial-temporal attention to ensure frame consistency. Yet succinct, our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model. We also have a better zero-shot shape-aware editing ability based on the text-to-video model [52]. Extensive experiments demonstrate our superior temporal consistency and editing capability than previous works.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/14ccb8bcceb6de10eda6ad08bec242a4f2946497

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Recently, benefiting from advancements in text-to-image research[2, 19, 29, 32, 34, 36], video generation (e.g., text-to-video, video editing)[4, 10, 12, 15–17, 23, 30, 40, 48, 52] has also achieved notable progress in terms of visual quality and diversity.”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, unknown): “This low-level vision task presents an additional challenge, as it requires high image ﬁdelity, which stands in contrast to the stochas-a r X i v : 2305 .”
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, background): “Such AI generative systems excel in various aspects, including rendering realistic and diverse appearances [20, 41, 16], powerful editing [9, 23], composition [1], generalization capabilities, and creating new art that satisﬁes people’s imagination [20, 41].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “FateZero [210] is the first framework for temporal-consistent zero-shot text-to-video editing using pre-trained text-to-image diffusion model.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “This results in a roughly uniﬁed appearance in the edited frames [53, 19, 5, 34].”
Video-P2P: Video Editing with Cross-Attention Control (2023, unknown): “Besides, there have been some concurrent works and fol-lowers after our work, such as [5, 9, 21, 30, 45, 49, 52].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “Besides pixel-wise error, we also consider the semantic consistency between every two frames, which is also used in previous video editing works [16,42].”
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, methodology): “For quantitative evaluation, we follow FateZero and Pix2Video to report Fram-Acc (CLIP-based frame-wise editing accuracy), Tmp-Con (CLIP-based cosine similarity between consecutive frames), Pixel-MSE (averaged mean-squared pixel error between aligned consecutive frames) in Table 2.”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “…and there are other related video generation tasks such as video-driven ( i.e ., video editing) [11, 12, 18, 29, 43, 58, 62, 67, 69, 76, 80, 82, 101, 116, 119, 121, 127–129], image-driven ( i.e ., image-to-video) [14, 16, 23, 27, 32, 76, 77, 88, 90, 100, 102, 106, 107, 122, 123],…”