Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

By Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz. arXiv.org, 2023

2,748 citations recorded by the source corpus.

We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1206b05eae5a06ba662ae79fb291b50e359c4f42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, background): “…videos from natural language inputs with impressive generalization capabilities (Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Podell et al., 2023; Dai et al., 2023; Esser et al., 2023; Blattmann et al., 2023b; Betker et al., 2023; Blattmann et al., 2023a; Singer et al., 2022).”
Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, neutral): “Nowadays, the cutting-edge diffusion models (Ho et al., 2022c; Blattmann et al., 2023a; OpenAI, 2024) and autoregressive models (Yan et al., 2021; Hong et al., 2023; Kondratyuk et al., 2024) have made remarkable breakthroughs in generating realistic and long-duration video through scaling of data…”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “Previous video training methods often involve joint training of images and videos with fixed number of frames (Singer et al., 2022; Blattmann et al., 2023).”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “Similar to prior work [10, 67], we found it important to shift the noise schedule towards high noise levels as we move from the pre-trained image diffusion model to our multi-view diffusion model that captures data of higher dimensionality.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, neutral): “In this study, we introduced DynamiCrafter , an effective framework for animating open-domain images by leveraging pre-trained video diffusion priors with the proposed dual-stream image injection mechanism and dedicated training paradigm.”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “With the aforementioned development on vision-language models, some concurrent works [7, 72, 75] also leverage these models for video captioning.”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, positive): “(1), where the target y is z 0 = E ( x ) following the pre-trained image-to-video diffusion model practices [4].”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, positive): “In this evaluation, we employ the ViT-B-32 clip model as the backbone, following the methodology out-lined in previous work (Blattmann et al., 2023) to ensure a fair comparison.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “MotionCtrl [13] is a camera-conditioned video diffusion model finetuned from SVD [17].”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “Despite the excellent generation performance of the recent open-source models such as Stable video diffusion [5], Open-sora [102], Open-sora-plan [46], Mochi-1 [79] and Allegro [104], their performance still falls far behind the closed-source state-of-the-art video generation models such as Sora…”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “( iv) SV3D [48]: an image-conditioned diffusion model based on Stable Video Diffusion [2] that generates an orbital video of an object, we only evaluate it on the novel view synthesis task since generating 3D meshes from its output is not straight-forward.”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, neutral): “While diffusion models [54, 18, 57, 30, 31] have revolutionized both synthesis and editing of images [10, 42, 41, 1, 43, 13, 9, 38] and videos [5, 4, 53, 2, 12, 19], their iterative nature remains a crucial shortcoming: At inference, a trained diffusion model usually requires dozens of network…”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Beyond the aforementioned steps, some video clips may contain multiple scenes, thus we introduce the Cascaded Cut Detector [30] to split multi-scene clips in S to achieve clip extraction, ensuring each contains only one scene.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, neutral): “Video diffusion models typically adopt bidirectional attention mechanisms to simultaneously denoise all video frames [3, 4, 6, 13, 23–26, 39, 64, 80, 83, 97].”
Genie: Generative Interactive Environments (2024, neutral): “Early signs indicate video generation will be yet another frontier, with recent results suggesting that such models may also benefit from scale (Blattmann et al., 2023a; Esser et al., 2023; Ho et al., 2022a; Hong et al., 2023).”
YOLOv12: Attention-Centric Real-Time Object Detectors (2025, neutral): “Interestingly, we find that this modification has only a slight impact on performance but significantly improves speed.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “For the Image-to-Video (I2V) setting, we use the Stable Video Diffusion (SVD) as the base video generator.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”