Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation

By Shuai Yang, Yifan Zhou, Ziwei Liu, Chen Change Loy. ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, 2023

333 citations recorded by the source corpus.

Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos. Code is available at our project page: https://www.mmlab-ntu.com/project/rerender/

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1e09b83fe064826a9a1ac61a7bdc00f26be41aee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Yu Qiao
2. Sangeek Hyun
3. Ziwei Liu
4. Jiwoo Chung
5. Jae-Pil Heo

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Luc Van Gool
3. Limin Wang
4. Ziwei Liu
5. Xiaoou Tang
6. Dacheng Tao
7. Xiaogang Wang
8. Andrew Zisserman

Citations in the corpus

• Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Many studies[10, 16, 17, 23, 26, 30, 40, 52, 54] explore the augmentation of inter-frame attention modeling on the foundation of text-to-image (T2I) models to achieve video generation.”
• Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer (2023, background): “Recent advances in Diffusion Models (DMs) have led to breakthroughs in various generative applications such as text-to-image synthesis [30, 33, 35] and image or video editing [2, 4, 6, 14, 19, 41, 48].”
• Video-P2P: Video Editing with Cross-Attention Control (2023, unknown): “Besides, there have been some concurrent works and fol-lowers after our work, such as [5, 9, 21, 30, 45, 49, 52].”
• VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “…other related video generation tasks such as video-driven ( i.e ., video editing) [11, 12, 18, 29, 43, 58, 62, 67, 69, 76, 80, 82, 101, 116, 119, 121, 127–129], image-driven ( i.e ., image-to-video) [14, 16, 23, 27, 32, 76, 77, 88, 90, 100, 102, 106, 107, 122, 123], personalized video…”