NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
By Junbin Xiao, Xindi Shang, Angela Yao, Tat-seng Chua. 2021
950 citations recorded by the source corpus.
We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git).
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Trevor Darrell
- Sergey Levine
- Andrew Zisserman
- Cordelia Schmid
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Jitendra Malik
- Cordelia Schmid
- Ming-Hsuan Yang
- Xiaogang Wang
In Question Answering & Reading Comprehension
Citations in the corpus
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
- PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Video QA tasks are treated as open-ended generation tasks; we report full-string accuracy (for MSR-VTT-QA and ActivityNet-QA) and WUPS metrics (NExT-QA) in [65, 59].”
- Video Instruction Tuning With Synthetic Data (2024, positive): “Since ShareGPT4Video lacks open-ended and multiple-choice questions, we supplemented with annotations from NExT-QA, PerceptionTest, and ActivityNet-QA.”
- LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “…3 Auto MSVD-QA [42] 13,157 10 Auto MSRVTT-QA [42] 72,821 15 Auto MVBench [17] 4,000 16 Auto Perception Test [29] 44,000 23 Auto&Manual NExT-QA [41] 52,044 44 Manual How2QA [18] 44,007 60 Manual ActivityNet-QA [45] 800 111 Manual CinePile [31] 303,828 160 Auto&Manual EgoSchema [24] 5,000 180…”
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “…VQA tasks, we utilize a mixture of publicly available datasets, including Kinetics-710 (Kay et al., 2017), SthSthv2 (Goyal et al., 2017), NExtQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), Tgif (Li et al., 2016), WebVidQA(Yang et al., 2021), RealworldQA (X. AI, 2024), and…”
- OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, positive): “909 between human and LLM evaluation (bootstrap CI=(0.883,0.928), N=9999), indicating excellent agreement with human judgement.”
- PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “…from Kinetics [13] and SthSthV2 [6], 450k caption data from Webvid [2], YouCook2 [47], TextVR [37] and VideoChat, 117 reasoning data from NextQA [38] and CLEVRER [42] and 109K questioning answering data annotated from Webvid, TGIF [20] and Ego4D [7], totaling 783k instructional tuning data.”
- Video Graph Transformer for Video Question Answering (2022, methodology): “The problem of answering questions that challenge real-world visual relation reasoning, especially the causal and temporal relations that feature video dynamics [20,59], is largely under-explored.”
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, background): “…Life ✗ MSRVTT-QA [Xu et al., 2017] Auto 2,990 72,821 15 Everyday Life ✗ ActivityNet-QA Yu et al. [2019] Human 800 8,000 180 Everyday Life ✗ NeXT-QA [Xiao et al., 2021] Human 1,000 8,564 44 Everyday Life ✗ MVBench [Wang et al., 2023] Auto 4,000 4,000 16 Life, Human Action, Movie ✗ EgoSchema…”
- LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “Note that the training set for LLaVA-Video includes six datasets: LLaVA-Video-178K, LLaVA-Hound (Zhang et al., 2024d), NExT-QA (Xiao et al., 2021), ActivityNet-QA (Yu et al., 2019), PerceptionTest (Pătrăucean et al., 2023), and image data from LLaVA-OneVision (Li et al., 2024c).”
- LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “For video data, we first integrate the multi-frame data from M4-Instruct, including NExT-QA [141] and ShareGPT4Video [21].”
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “…reasoning capabilities of VideoLLaMA3, we conduct evaluations on the following tasks: (1) Temporal Perception and Reasoning tasks, including TempCompass [134] and NextQA [135]; and (2) Temporal Sentence Grounding task on Charades-STA [108] benchmark, with mean Intersection over Union (mIoU) metric.”
- ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “This collection includes 13K conversational data from VideoChatGPT [42] and 140K question-answer pairs, with 45K data points from CLEVRER [64], 8K from EGO-QA [18], 34K from NextQA [61], and 53K from TGIF-Transition [31].”
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “The NextQA dataset has two sub-datasets: video question answering with multiple choice and open-domain video question answering.”
- TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “…(Puatruaucean et al., 2023) ✓ ✓ ✗ VITATECS (Li et al., 2023e) ✓ ✗ ✓ Video LLM Benchmarks SEEDBench (Li et al., 2023a) ✗ ✗ ✗ Video-Bench (Ning et al., 2023) ✗ ✗ ✓ VLM-Eval (Li et al., 2023f) ✗ ✓ ✓ AutoEval-Video (Chen et al., 2023) ✓ ✗ ✓ MVBench (Li et al., 2023d) ✓ ✗ ✓ Table 1: Comparison with…”
- MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “There are specialized benchmarks for temporal perception [49; 42], action understanding [42; 41], video classification [14], video reasoning [45; 44], and video captioning [47; 30].”
- Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, background): “Using multimodal inputs does not increase the performance significantly, as summarised in [35]), so that fine-tuning is no longer necessary.”
- Revisiting the “Video” in Video-Language Understanding (2022, methodology): “We find that even recent benchmarks that explicitly design for temporal and causal understanding (e.g., [54]), can have a non-trivial sub-set of questions answerable by simple single-frame event recognition.”
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, neutral): “The video training data contains a subset of VideoChat2-IT (Li et al., 2024b), which includes TextVR (Wu et al., 2025), Youcook2 (Zhou et al., 2018), Kinetics-710 (Kay et al., 2017), NExTQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), TGIF (Li et al., 2016), WebVidQA (Yang et a”