TVQA: Localized, Compositional Video Question Answering
By Jie Lei, Licheng Yu, Mohit Bansal, Tamara L. Berg. 2018
791 citations recorded by the source corpus.
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Zhe Gan
- Yejin Choi
- Percy Liang
- Caiming Xiong
- Yoshua Bengio
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Cordelia Schmid
- Yi Yang
- Ming-Hsuan Yang
- Jitendra Malik
In Question Answering & Reading Comprehension
Citations in the corpus
- STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Various benchmarks [11, 44, 20] have been constructed by asking annotators to provide designed questions [1, 44] or synthesizing questions with pre-defined templates [15, 24, 14, 6].”
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[74] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg.”
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Table 6 shows the performance of the representative approaches used in Table 2 on TVQA, How2QA, TVC, and YC2C.”
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “The tasks we report on are: TVQA [53], TVQA(+) [54], VLEP [55], MSRVTT-QA [105], MSRVTTMultichoice [109], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [97, 81], ActivityNetQA [110, 37], TGIFQA [44], and DramaQA [18]5.”
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Regarding visual feature representations, motion feature are important but naively concatenating appearance and motion features usually results in sub-optimal results (refer to EVQA [2], PSAC+ [26] and STVQA [27]).”
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Joint visual-text models have become increasingly popular as they enable a wide suite of downstream tasks, including text-to-visual retrieval [29, 31, 36, 59], visual captioning [24, 58, 66], and visual question and answering [4, 27].”
- Unifying Vision-and-Language Tasks via Text Generation (2021, background): “, 2018b) in a wide range of discriminative (Goyal et al., 2019; Hudson & Manning, 2019; Lei et al., 2018; Mao et al., 2016; Xu et al., 2016; Zhou et al., 2018) and generative tasks (Chen et al.”
- ActBERT: Learning Global-Local Video-Text Representations (2020, background): “, video question answering [36, 10, 18, 54], video captioning [46, 52], textvideo retrieval [47, 41, 25], video grounding [50].”
- Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “TV Dataset (Lei et al., 2018) was built on 6 popular TV shows across 3 genres: medical 6Some existing Video QA tasks require localizing ‘frames of interest’ for the question, e.g., TVQA+ (Lei et al., 2019). dramas, sitcoms and crime shows.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Alpaca: Five words that rhyme with shock are: rock, pop, shock, cook, and snock.”
- From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “While past work has asked humans to write several counterfactual choices for each correct answer [75, 46], this process is expensive.”
- A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “, 2018) Visual Question Answering TVQA (Lei et al., 2018) Video Question Answering 4 4”
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, background): “Popular video-andlanguage tasks include text-to-video retrieval [75, 28, 54], video captioning [75, 82, 28, 54, 40], video question answering [74, 23, 33], and moment retrieval [2, 18, 35].”
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “The VCR dataset (Zellers (Tapaswi et al., 2016) × × × MarioQA (Mun et al., 2017) × × × TVQA (Lei et al., 2018) × × × × Social-IQ (Zadeh et al., 2019) × × × ×”
- Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “This technique is more useful for videos that are longer [40] and with more complex structures such as movies [33] and TV programs [17] with extra accompanying channels such as speech or subtitles.”
- TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “We also evaluate the two most representative non-neural baselines from [19], i.”
- Support-set bottlenecks for video-text representation learning (2020, background): “0 JSFusion (Yu et al., 2018) 10 .”