Phenaki: Variable Length Video Generation From Open Domain Textual Description

By Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, Dumitru Erhan. 2022

572 citations recorded by the source corpus.

We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or a story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts. In addition, compared to the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

MusicLM: Generating Music From Text (2023, methodology): “Modeling sequences of discrete tokens autoregressively has proven to be a powerful approach in natural language processing (Brown et al., 2020; Cohen et al., 2022) and image or video generation (Esser et al., 2021; Ramesh et al., 2021; Yu et al., 2022; Villegas et al., 2022).”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, background): “Early attempts to pretrain and scale Transformers to generate videos from text have shown great promise, such as CogVideo (Hong et al., 2022) and Phenaki (Villegas et al., 2022).”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “Some work [67] has used causal attention in self-attention layers of a video VAE, but we did not find any image VAE work uses causal self-attention.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “…of available compute resources and datasets better suited for generative modeling such as WebVid-10M [7], more competitive approaches have been proposed recently, mainly based on well-scalable, explicit likelihood-based approaches such as diffusion [41, 42, 82] and autoregressive models [94].”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “The Masked LM (MLM) is another type of language model for visual generation, such as: MaskGIT (Chang et al., 2022), MAGVIT (Yu et al., 2023a), Phenaki (Villegas et al., 2022), and MUSE (Chang et al., 2023), among others.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “Video diffusion models typically adopt bidirectional attention mechanisms to simultaneously denoise all video frames [3, 4, 6, 13, 23–26, 39, 64, 80, 83, 97].”
ChatGPT is not all you need. A State of the Art Review of large Generative AI models (2023, methodology): “Phenaki : This model has been made by Google Research, and it is capable of performing realistic video synthesis, given a sequence of textual prompts [34].”
MAGVIT: Masked Generative Video Transformer (2022, background): “As shown in recent works [31, 50, 64], training such models requires large, and sometimes non-public, datasets of paired texts and images.”
Learning Universal Policies via Text-Guided Video Generation (2023, background): “However, the desired planner departs from the typical setting in text-to-video models (Villegas et al., 2022; Ho et al., 2022a) which normally generate unconstrained videos given a text description.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Both families have been extended to text-to-video [36, 37, 65, 73] using paired data.”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “To reconcile the per-segment SSR predictions, { Φ( J i ) } Ni =1 , we define the result of the denoising step to be the solution of the optimization problem arg min The solution to this problem is given by linearly combining the predictions over overlapping windows.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[90] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan.”
Pix2Video: Video Editing using Image Diffusion (2023, methodology): “With the success of text-to-image generation models, there has been recent attempts in text-to-video generation models using architectures such as transformers [20, 52, 59] or diffusion models [15, 18, 45, 60].”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “To replicate this success in Text-to-Video (T2V) generation, recent works [43, 16, 19, 54, 48] have extended spatial-only T2I generation models to the spatio-temporal domain.”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, unknown): “Diffusion Models: Diffusion models [35, 69] have recently gained signiﬁcant attention from the research community due to their ability to generate high-ﬁdelity and diverse content like images [66, 54, 24], videos [68, 34, 77], 3D [58, 49], and audio [43, 51] from various input modalities like text.”