VBench: Comprehensive Benchmark Suite for Video Generative Models

By Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qin Jin, Nattapol Chanpaisit and 6 more. Computer Vision and Pattern Recognition, 2023

1,673 citations recorded by the source corpus.

Video generation has witnessed significant advance-ments, yet evaluating these models remains a challenge. A comprehensive evaluation benchmark for video generation is indispensable for two reasons: 1) Existing metrics do not fully align with human perceptions; 2) An ideal eval-uation system should provide insights to inform future de-velopments of video generation. To this end, we present VBench, a comprehensive benchmark suite that dissects “video generation quality” into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. VBench has three appealing proper-ties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation (e.g., subject identity in-consistency, motion smoothness, temporal flickering, and spatial relationship, etc.). The evaluation metrics with fine-grained levels reveal individual models' strengths and weaknesses. 2) Human Alignment: We also provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception, for each evaluation dimension respectively. 3) Valuable Insights: We look into current models' ability across various evaluation dimensions, and various content types. We also investi-gate the gaps between video and image generation models. We will open-source VBench, including all prompts, evaluation methods, generated videos, and human preference an-notations, and also include more video generation models in VBench to drive forward the field of video generation.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4e9a8141da2a8c603722b07d096109207f8e0b66

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VACE: All-in-One Video Creation and Editing (2025, neutral): “VBench [27] and VBench++ [28] have established a precise evaluation framework for text-to-video and image-to-video tasks through an extensive assessment suite and dimensional de-sign.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “In addition, we leverage a novel video generation evaluation benchmark suit VBench (Huang et al., 2023) to thoroughly analyze the generated video quality of LaVie and state-of-the-art.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “We adopt VBench [31] and user preference study to evaluate both visual quality and semantic alignment.”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “The used benchmark is VBench [33], a comprehensive toolkit for evaluating video generation performance, which assesses the quality and semantic capabilities of each [39] and Gen-3 [75], it outperforms the majority of open-source text-to-video models.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “…we successfully integrate the benefits of both pixel and latent-based VDMs within a cohesive framework, named Show-1, which achieves state-of-the-art performance on popular video generation benchmarks including UCF101 (Soomro et al., 2012), MSR-VTT (Xu et al., 2016) and VBench (Huang et al., 2023).”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “To evaluate the text-to-video generation, we employed several metrics from VBench (Huang et al., 2024): Human Action , Scene , Dynamic Degree , Multiple Objects , and Appearance Style .”
Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, positive): “It shares a similar structure with MAGVIT-v2 (Yu et al., 2024) and is trained from scratch on the WebVid-10M dataset (Bain et al., 2021).”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “Our method is evaluated VBench [26], a benchmark for video generation with 16 metrics designed to systematically assess motion quality and semantic alignment.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “AnimateDiff can be integrated with various T2I LoRAs or base models across (Chen et al., 2023a) and SVD (Blattmann et al., 2023a) as (Radford et al., 2021), and Frame Consistency (FC) (Huang et al., 2023) to evaluate the video appearance quality.”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, neutral): “• A fish is swimming in the aquarium tank.”