Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

By Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis. Computer Vision and Pattern Recognition, 2023

1,730 citations recorded by the source corpus.

Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. Here, we apply the LDM paradigm to high-resolution video generation, a particularly resource-intensive task. We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and finetuning on encoded image sequences, i.e., videos. Similarly, we temporally align diffusion model upsamplers, turning them into temporally consistent video super resolution models. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling. In particular, we validate our Video LDM on real driving videos of resolution $512 \times 1024$, achieving state-of-the-art performance. Furthermore, our approach can easily leverage off-the-shelf pretrained image LDMs, as we only need to train a temporal alignment model in that case. Doing so, we turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to $1280 \times 2048$. We show that the temporal layers trained in this way generalize to different finetuned text-to-image LDMs. Utilizing this property, we show the first results for personalized text-to-video generation, opening exciting directions for future content creation. Project page: https://nv-tlabs.github.io/VideoLDM/

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f5a0c57f90c6abe31482e9f320ccac5ee789b135

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Following the protocols in [76], we report CLIPSIM, IS, FID, and FVD metrics.”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Similar to video diffusion models (Ho et al., 2022; Singer et al., 2022; Blattmann et al., 2023; Zhou et al., 2022), we would like to adapt the attention layers to model the cross-view dependency while keeping the remaining network as a 2D model that only operates within a single image.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “Most related works make use of a pretrained text-to-image model and insert temporal mixing layers of various forms [1, 9, 29, 31, 32] into the pretrained architecture.”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “Most existing methods [14, 23, 26, 30, 48, 63] follow a logic to obtain video models, i.e., extending a text-to-image (T2I) backbone to a video model by adding temporal modules and then training it with videos.”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “For example, they only evaluate the performance using FVD [56] (LVDM [20], MagicVideo [69], Align Your Latent [8]), IS [46] (Align Your Latent [8]), CLIP similarity [43] (Gen1 [16], Imagen Video [22], Make-A-Video [49]), or user studies to show the performance level.”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “We will open-source our VBench and encourage more T2V models to participate in the evaluation, including but not limited to [1–4, 8, 124], so that we can provide more informed insights into the current state of T2V, and provide more annotated data on T2V generation results generated by different…”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, background): “Diffusion models [14, 47] have recently shown to be powerful generative models that can generate a wide variety of images [3, 39, 40] and videos [2, 12, 51] by iteratively denoising a noise sample.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “…visual encoders, such as the VAE encoder from Stable Diffusion [19], is advisable for most researchers while Sora ’s team is expected to train their own compression network with a decoder (the video generator) from scratch via the manner employed in training latent diffusion models [19, 35, 36].”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “The latter is often referred to as latent diffusion models (LDMs) and has become the standard approach for modeling high-dimensional data such as videos [4, 25, 96, 109, 111].”
Genie: Generative Interactive Environments (2024, methodology): “Video models Our work is related to video models , which typically condition on initial frames (or text) and predict the remaining frames in a video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al.,…”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “Among them, Align-Your-Latents (Blattmann et al., 2023b) efficiently turns a text-to-image (T2I) model into a video generator by aligning independently sampled noise maps.”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “Regarding text-to-video (T2V) models, Make-A-Video [45] and Imagen Video [27] are cascaded models, while most other works, such as LVDM [24], Magic Video [63], Mod-elScope [50], and Align your Latents [10], are SD-based models.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “Text-to-image and text-to-video generation has garnered significant rapid advancements using both language models (Yu et al., 2023b; Chang et al., 2023) and diffusion models (Ho et al., 2022a; Blattmann et al., 2023; Singer et al., 2022; Ge et al., 2023; Ramesh et al., 2022).”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “Video diffusion models typically adopt bidirectional attention mechanisms to simultaneously denoise all video frames [3, 4, 6, 13, 23–26, 39, 64, 80, 83, 97].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “A similar idea is adopted by several recent works [46, 69, 4].”
Learning Interactive Real-World Simulators (2023, background): “Internet-scale generative models can synthesize realistic images and videos [44, 3, 45, 46, 47], but have mostly been applied to generative media [48] as opposed to empowering sophisticated agents capable of multi-turn interactions.”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, positive): “In diffusion-based video generation models, latent diffusion models [5] are frequently employed to compress the video for operation in a lower-dimensional space.”