TVQA+: Spatio-Temporal Grounding for Video Question Answering

By Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal. 2019

281 citations recorded by the source corpus.

We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to visual concepts in questions and answers. We name this augmented version as TVQA+. We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our TVQA+ dataset can contribute to the question answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal attention visualizations.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, neutral): “Dataset Real-World Situation Diagnostic Interaction Sequence Prediction Feasibility Videos Abstraction Reasoning VQA [1] ✗ ✗ ✗ ✓ ✗ ✗ ✗ VCR [49] ✗ ✗ ✗ ✓ ✓ ✗ ✗ GQA [18] ✗ ✓ ✗ ✗ ✗ ✗ ✗ CLEVR [21] ✗ ✗ ✓ ✗ ✗ ✗ ✗ COG [46] ✗ ✗ ✓ ✗ ✗ ✗ ✗ CLEVRER [47] ✗ ✗ ✓ ✓ ✓ ✓ ✗ TGIF-QA [19] ✓ ✗ ✗ ✓ ✓ ✗ ✗ MovieQA [40] ✓ ✗ ”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “The tasks we report on are: TVQA [53], TVQA(+) [54], VLEP [55], MSRVTT-QA [105], MSRVTTMultichoice [109], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [97, 81], ActivityNetQA [110, 37], TGIFQA [44], and DramaQA [18]5.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “In contrast to some previous works using additional video features such as subtitles [12, 36, 37, 43, 44, 46, 74, 80, 87], our video representation is exclusively based on visual information, as we focus on the visual understanding of videos.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “Later on, Lei et al. (2019) augmented TVQA with frame-level bounding box annotations for spatial-temporal video QA, and introduced the STAGE framework to jointly localize moments, ground objects, and answer questions.”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, background): “A wide range of tasks based on real-life videos have been designed to test such ability, including text-to-video retrieval [75, 28, 54], video captioning [54, 75, 82], video question answering [74, 23, 33, 34], and video moment retrieval [2, 18, 35].”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Popular tasks include text-based video retrieval [70, 28, 53, 35, 36], video moment retrieval [7, 21, 28, 35, 36], video question answering [68, 25, 32, 33], and video captioning [53, 70, 74, 35].”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, unknown): “While QA based on a single image, i.e., VQA, has been well explored [17, 2, 49, 1, 35, 36], the ﬁeld of VideoQA is now picking up intense momentum [62, 22, 7, 30, 28].”
BERT Representations for Video Question Answering (2020, methodology): “For comparison, we report results on TVQA [21], STAGE [22] and MDAM [17].”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “Some prior work has explored commonsense reasoning tasks posed over videos, which usually have more input available than a single frame [12,13,19,20,30,31,33,63,74,78] (inter alia).”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “Meanwhile, there are many remaining challenges, such as resolving the performance gap between zero-shot and training with in-domain non-English annotations; as well as techniques to transfer varieties of vision-language models (e.g., VQA (Goyal et al., 2017), TVQA (Lei et al., 2020)) or visually-enh”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, background): “For moment retrieval, there are work that try to score generated moment proposals [13, 36, 4, 49, 47, 41], predict moment start-end indices [18, 9, 19, 21, 45, 48] or regress moment coordinates [7].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “, video question answering [37, 38], video captioning [34, 98], and human action recognition [36, 62, 63]).”
Video-Text Pre-training with Learned Regions (2021, background): “It enables a series of downstream video-text tasks, such as video-text retrieval [32, 56, 73], video question answering [25,35,36], and video captioning [56,73,78].”