Towards Accurate Generative Models of Video: A New Metric & Challenges

By Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, Sylvain Gelly. 2018

1,343 citations recorded by the source corpus.

Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent attempts at formulating generative models of video have had some success, current progress is hampered by (1) the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples, and (2) the wide gap between purely synthetic video data sets and challenging real-world data sets in terms of complexity. To this extent we propose Fr\'{e}chet Video Distance (FVD), a new metric for generative models of video, and StarCraft 2 Videos (SCV), a benchmark of game play from custom starcraft 2 scenarios that challenge the current capabilities of generative models of video. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos, and provide initial benchmark results on SCV.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Scaling Autoregressive Video Models (2019, methodology): “In terms of perceptual quality, every incarnation of our model obtains a lower (better) FVD score compared to all models evaluated by Unterthiner et al. (2018), which notably includes adversarial networks with no guarantees of covering the full empirical distribution.”
Scalable Adaptive Computation for Iterative Generation (2022, methodology): “For evaluation, we follow common practice (Ho et al., 2022b) and use FVD (Unterthiner et al., 2018) and Inception Scores computed on 50K samples, with 400 or 1000 steps of DDPM.”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “For example, they only evaluate the performance using FVD [56] (LVDM [20], MagicVideo [69], Align Your Latent [8]), IS [46] (Align Your Latent [8]), CLIP similarity [43] (Gen1 [16], Imagen Video [22], Make-A-Video [49]), or user studies to show the performance level.”
Adversarial Video Generation on Complex Datasets (2019, methodology): “The FVD of this video synthesis model is notably better.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “FVD (Unterthiner et al., 2018) is used as our primary evaluation metric.”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, neutral): “The conditioning signal C contains this first frame followed by blank frames for the rest of the video.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “To evaluate our models, we use frame-wise Fr ´ echet Inception Distance (FID) [26] as well as Fr ´ echet Video Distance (FVD) [87].”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, neutral): “Video-level evaluation uses FVD[44] metrics.”
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (2022, methodology): “Following Rakhimov et al. [17], Yu et al. [37], we use Fréchet Video Distance (FVD) [27] and Inception score (IS) [21] as metrics in the evaluation.”
Hierarchical Style-based Networks for Motion Synthesis (2020, background): “quences than these two baselines (rst two rows). The proportion of results of the other models classied as positive (third row) is lower. Distribution Similarity Evaluation. Inspired from FVD score [30] we measure the distribution similarity between the original and generated sequences with motion f”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, methodology): “Fréchet Video Distance (FVD): We evaluate VideoFlow using the recently proposed Fréchet Video Distance (FVD) metric (Unterthiner et al., 2018), an adaptation of the Fréchet Inception Distance (FID) metric (Heusel et al., 2017) for video generation.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, positive): “We filter 5369 valid samples from the validation set to conduct FID [47] and FVD [115] evaluation.”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “We achieve FVD scores of 555.5 and 2252.7 respectively.”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, methodology): “StarGAN-v2 (Brown Hair) MUNIT (Eyeglasses) Metrics Original Video Original Video Reference Label Reference Label FVD (↓) 323.71 244.58 295.74 232.63 204.12 158.87 FID (↓) 77.26 64.82 89.07 69.68 30.65 31.23 test set.”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “Other methods [97], [150], [151] also reported results using: LPIPS [230] as a perceptual metric comparing CNN features, or Frchet Video Distance (FVD) [231] to measure sample realism by comparing underlying distributions of predictions and ground truth.”
Predicting Video with VQVAE (2021, methodology): “One proposed metric used on larger datasets such as Kinetics-600 is the Fréchet Video Distance [51].”