Perception Test: A Diagnostic Benchmark for Multimodal Video Models

By Viorica Puatruaucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yezhou Yang and 14 more. Neural Information Processing Systems, 2023

350 citations recorded by the source corpus.

We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baseline code, and challenge server are available at https://github.com/deepmind/perception_test

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/42585728685d7b2567e62b03463d1520f1bbe47e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “Furthermore, including the PerceptionTest dataset enhanced its associated task.”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “For the MC-VQA task, we select EgoSchema (Mangalam et al., 2024), Perception-Test (Patraucean et al., 2024), MV-Bench (Li et al., 2023c), and VideoMME (Fu et al., 2024).”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “We further expand the training set by including datasets from PerceptionTestQA [Patraucean et al., 2024], TVQA [Lei et al., 2018], NTU-RGB-D [Liu et al., 2020], and EgotaskQA [Grauman et al., 2022], along with grounding datasets based on DiDeMo [Anne Hendricks et al., 2017b] and COCO [Lin et al.,…”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…& zh) [130, 131], EgoTaskQA (en) [99], NTU RGB+D (en) [152], CLEVRER (en) [276], LLaVA-Video (en) [307], FineVideo (en) [67], PerceptionTest (en) [193], HiREST (en) [291], STAR (en) [259], General QA EgoSchema (en) [175], ScanQA (en) [10], LSMDC (en) [201] GUI GUI-World (en) [24] Type: Text…”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “In contrast, the LongVILA model (right) demonstrated enhanced performance across a range of frame counts and depths.”
Video Instruction Tuning With Synthetic Data (2024, positive): “Furthermore, including the PerceptionTest dataset significantly enhanced its associated task.”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “We observe that V-JEPA 2 exhibits competitive performance in the frozen setup, outperforming DINOv2, SigLIP, and Perception Encoder (PE) in all of the tested benchmarks (Table 6) except PerceptionTest where V-JEPA 2 slightly underperforms SigLIP and PE.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “We assess VideoLLaMA3’s general video understanding capabilities through established benchmarks: (1) Multi-Choice Video Question Answering (MC-VQA) tasks, including MVBench [26], VideoMME [126], EgoSchema [127], and Perception-Test [128].”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “Num QA Avg sec Open-domain Multi-type Annotation TGIF-QA [14] 165,165 3 Auto MSVD-QA [42] 13,157 10 Auto MSRVTT-QA [42] 72,821 15 Auto MVBench [17] 4,000 16 Auto Perception Test [29] 44,000 23 Auto&Manual NExT-QA [41] 52,044 44 Manual How2QA [18] 44,007 60 Manual ActivityNet-QA [45] 800 111 Manual C”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “…action-related QA pairs derived from ActivityNet dataset, EgoSchema [98] and MLVU [169] focusing on long video understanding, PerceptionTest [115] designed to evaluate the perception skills, VideoMME [29] and NeXTQA [141] containing diverse video domains and durations (from minutes to…”