Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators

By L. Khachatryan, A. Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, Humphrey Shi. IEEE International Conference on Computer Vision, 2023

848 citations recorded by the source corpus.

Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets. In this paper, we introduce a new task, zero-shot text-to-video generation, and propose a low-cost approach (without any training or optimization) by leveraging the power of existing text-to-image synthesis methods (e.g. Stable Diffusion), making them suitable for the video domain. Our key modifications include (i) enriching the latent codes of the generated frames with motion dynamics to keep the global scene and the background time consistent; and (ii) reprogramming frame-level self-attention using a new cross-frame attention of each frame on the first frame, to preserve the context, appearance, and identity of the foreground object. Experiments show that this leads to low overhead, yet high-quality and remarkably consistent video generation. Moreover, our approach is not limited to text-to-video synthesis but is also applicable to other tasks such as conditional and content-specialized video generation, and Video Instruct-Pix2Pix, i.e., instruction-guided video editing. As experiments show, our method performs comparably or sometimes better than recent approaches, despite not being trained on additional video data. Our code is publicly available at: https://github.com/Picsart-AI-Research/Text2Video-Zero.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/923a03032014a12c4e8b26511c0394e1b915fe74

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “In contrast to works that only train temporal layers [9, 32] or are completely training-free [52, 114], we finetune the full model.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “The edits of Text2Video-Zero [19] (ﬁfth row) suffer from severe jittering as this method relies heavily on the extended attention mechanism to implicitly encourage consistency.”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, methodology): “Text2Video-Zero [15] introduces a training-free method to animate a pre-trained T2I model via latent wrapping given a predeﬁned afﬁne matrix.”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, methodology): “Rather than synthesizing all frames independently, [15,40] enhance appearance coherence by replacing original self-attention with the sparser cross-frame attention.”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Recently, benefiting from advancements in text-to-image research[2, 19, 29, 32, 34, 36], video generation (e.g., text-to-video, video editing)[4, 10, 12, 15–17, 23, 30, 40, 48, 52] has also achieved notable progress in terms of visual quality and diversity.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Recent research has demonstrated the potential of employing diffusion models for video synthesis [6, 18, 23, 28, 32, 68].”
VACE: All-in-One Video Creation and Editing (2025, neutral): “…3) For controllable task, we use Control-A-Video [10], VideoComposer [68], and ControlVideo [79] under depth conditions, and compare Text2Video-Zero [31], ControlVideo [79], and Follow-Your-Pose [40] under pose conditions, as well as FLAT-17196 while being comparable to the metrics of Vidu 2.0.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Similar to adding controls in T2I [48, 61, 88, 93], introducing control signals in T2V, such as structure [17, 83], pose [44, 94], and Canny edge [39], has been increasingly receiving much attention.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Zero-shot video generation results of Text2Video-Zero [129].”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “To provide enhanced guidance, some works adopt other signals, e.g. , depth/skeleton sequence, to precisely control the scene/human motion in the generated videos (Guo et al., 2023a; Chen et al., 2023c; Zhang et al., 2023c; Khachatryan et al., 2023; Hu et al., 2023; Xu et al., 2023).”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, neutral): “…and unconditional video synthesis (Voleti et al., 2022; Harvey et al., 2022; Zhou et al., 2022; Wu et al., 2022b; Blattmann et al., 2023b; Khachatryan et al., 2023; H¨oppe et al., 2022; Voleti et al., 2022; Yang et al., 2022; Nikankin et al., 2022; Luo et al., 2023; An et al., 2023; Wang…”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Current text-to-video generation methods can be divided into two categories from architecture design: UNet based methods [19, 20, 7, 21, 22, 23, 24], and DiT based methods [6, 8, 5].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Recently, driven by the tremendous success of applying the diffusion model to image synthesis, multiple works have proposed to explore diffusion models for conditional and unconditional video synthesis [57, 12, 70, 61, 4, 22, 19, 57, 65, 33, 28, 1, 59].”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “…task in video generation, and there are other related video generation tasks such as video-driven ( i.e ., video editing) [11, 12, 18, 29, 43, 58, 62, 67, 69, 76, 80, 82, 101, 116, 119, 121, 127–129], image-driven ( i.e ., image-to-video) [14, 16, 23, 27, 32, 76, 77, 88, 90, 100, 102, 106,…”
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, methodology): “Similar to other zero-shot video editing methods [3, 20], we replace self-attention layers in the U-Net with crossframe attention layers to regularize the global style of I ′ i to match that of I ′ 1 and I ′ i−1.”