EvalCrafter: Benchmarking and Evaluating Large Video Generation Models

By Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond H. Chan, Ying Shan. Computer Vision and Pattern Recognition, 2023

347 citations recorded by the source corpus.

The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos. However, these methods often use a few metrics, e.g., FVD [56] or IS [45], to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a novel framework and pipeline for exhaustively evaluating the performance of the generated videos. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation, which is based on an analysis of real-world user data and generated with the assistance of a large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics. To obtain the finalleaderboard of the models, we further fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed human alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/66d927fdb6c2774131960c75275546fd5ee3dd72

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “For each model, we employ a consistent set of 700 prompts from [35] to generate videos.”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “For video-text alignment, we use the EvalCrafter [50] benchmark with 1-5 human Likert scales collected by [81].”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “…small proportion of input videos with text describing camera motion, our model can use short text prompts to apply a range of camera motions to image-to-video and text-to-video generations (see Figure 13), which has been noted to be difficult for many state-of-the-art video generation models [40].”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “Our work differs from concurrent research [70, 71] in three key ways: 1) We have created 16 distinct evaluation dimensions, each with specialized prompts for precise assessment; 2) We have empirically validated that every dimension aligns closely with human perception; 3) Our multi-dimensional and…”