ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
By Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, Dacheng Tao. 2019
754 citations recorded by the source corpus.
Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA performance, especially for long videos.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Trevor Darrell
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Jitendra Malik
- Ming-Hsuan Yang
- Xiaogang Wang
- Andrew Zisserman
In Evaluation & Benchmarks
- Ali Farhadi
- Yejin Choi
- Jitendra Malik
- Li Fei-Fei
- Cordelia Schmid
- Antonio Torralba
- Andrew Zisserman
- Yu Qiao
In Question Answering & Reading Comprehension
- Tat-seng Chua
- Yueting Zhuang
- Cordelia Schmid
- Junbin Xiao
- Devshree Patel
- Ratnam Parikh
- Yesha Shastri
- Zhou Zhao
Citations in the corpus
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “We report results on 12 video reasoning tasks: TVQA [64], TVQA(+) [65], VLEP [66], MSRVTT-QA [117], MSRVTT-Multichoice [121], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [110, 92], ActivityNetQA [122, 45], TGIFQA [52], and DramaQA [23].”
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Unlike previous works [18, 56] which focus on reasoning temporal actions of a single object in a question, we emphasize more on object interactions.”
- Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [11, 16, 17, 21, 27, 33, 38, 43, 46, 60, 69, 74, 84, 90, 91, 92, 93] are limited in size, as the largest, TGIF-QA [33], contains only 72K annotated clips (see Figure 2 for more details).”
- Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, methodology): “Different from Image QA, Video QA methods (Jang et al. 2017; Xu et al. 2017; Gao et al. 2018; Li et al. 2019; Yu et al. 2019; Fan et al. 2019) introduce extra temporal reasoning modules to answer questions which involve changing on temporal dimension like action or state transition.”
- Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.”
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “We test our method on four common video question answering benchmarks, TGIF-QA (Jang et al., 2017), MSVD-QA (Xu et al., 2017), MSRVTT-QA (Xu et al., 2017) and ActivityNet-QA (Yu et al., 2019), where video-question pairs are massively disproportional in length.”
- OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, positive): “Ultimately, we find a large gap be-tween the best models (GPT-4V at 49.6”
- Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, background): “Existing real-world benchmarks rely on one of the following data sources: (i) Videos collected from the web or repositories like Youtube, e.g. Kinetics [36], ActivityNet [9], VGGSound [11], HVU [18], ActivityNet-QA [56], tGIFQA [33]; (ii) Videos collected on demand, filmed by volunteers doing arbitr”
- Span-based Localizing Network for Natural Language Video Localization (2020, background): “As an important vision-language understanding task, NLVL involves both computer vision and natural language processing techniques (Kr-ishna et al., 2017; Hendricks et al., 2017; Gao et al., 2018; Le et al., 2019; Yu et al., 2019).”
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, methodology): “MSVD-QA [Xu et al., 2017] Auto 520 13,157 10 Everyday Life ✗ MSRVTT-QA [Xu et al., 2017] Auto 2,990 72,821 15 Everyday Life ✗ ActivityNet-QA Yu et al. [2019] Human 800 8,000 180 Everyday Life ✗ NeXT-QA [Xiao et al., 2021] Human 1,000 8,564 44 Everyday Life ✗ MVBench [Wang et al., 2023] Auto 4,000 4,”
- MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “There are specialized benchmarks for temporal perception [49; 42], action understanding [42; 41], video classification [14], video reasoning [45; 44], and video captioning [47; 30].”
- PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “First, the open-ended Video Question Answer (VideoQA) includes MSVD-QA [39], MSRVTT-QA [39], ActivityQA [43], and TGIF QA [20].”
- Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, methodology): “We evaluate methods for solving VQA tasks on ActivityNet-QA (Yu et al., 2019).”
- ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “…approximately 15,000 QA pairs spanning 10 evaluation dimensions from 13 existing data sources, such as MSVD-QA [62], MSRVTT-QA [62], Activitynet-QA [67], etc. MVBench [30] is designed to challenge LVLMs with video tasks that cannot be effectively resolved by single-frame reliance, featuring 4,000…”
- VideoMamba: State Space Model for Efficient Video Understanding (2024, neutral): “Beyond action recognition, large-scale video-text datasets [10, 13, 58, 84, 87, 89] extend video understanding into multi-modality tasks like video captioning, retrieval, and question answering.”
- LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “…NExT-QA [41] 52,044 44 Manual How2QA [18] 44,007 60 Manual ActivityNet-QA [45] 800 111 Manual CinePile [31] 303,828 160 Auto&Manual EgoSchema [24] 5,000 180 Auto&Manual MovieQA [36] 6,462 203 Manual LongVideoBench [40] 6,678 473 Manual MovieChat-1K [34] 13,000 564 Manual MoVQA [49] 21,953 992…”
- Video Instruction Tuning With Synthetic Data (2024, positive): “For full evaluation, we consdier 11 video benchmarks. conducted tests across various video captioning , video open-ended question-answering and video multiple-choice question-answering benchmarks, including ActivityNet-QA (Yu et al., 2019), which features human-annotated action-related QA pairs from”
- LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “Note that the training set for LLaVA-Video includes six datasets: LLaVA-Video-178K, LLaVA-Hound (Zhang et al., 2024d), NExT-QA (Xiao et al., 2021), ActivityNet-QA (Yu et al., 2019), PerceptionTest (Pătrăucean et al., 2023), and image data from LLaVA-OneVision (Li et al., 2024c).”
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, positive): “In contrast, the LongVILA model (right) demonstrated enhanced performance across a range of frame counts and depths.”