TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
By Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, Gunhee Kim. 2017
700 citations recorded by the source corpus.
Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Cordelia Schmid
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Ming-Hsuan Yang
- Jitendra Malik
In Question Answering & Reading Comprehension
- Chenyou Fan
- Mohit Bansal
- Tat-seng Chua
- Jie Lei
- Junbin Xiao
- Devshree Patel
- Ratnam Parikh
- Yesha Shastri
Citations in the corpus
- STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Various benchmarks [11, 44, 20] have been constructed by asking annotators to provide designed questions [1, 44] or synthesizing questions with pre-defined templates [15, 24, 14, 6].”
- VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Choice TGIF-Action [6] 5 TGIF-Transition [6] 5 MSRVTT-MC [7] 5 LSMDC-MC [3] 5”
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “We report results on 12 video reasoning tasks: TVQA [64], TVQA(+) [65], VLEP [66], MSRVTT-QA [117], MSRVTT-Multichoice [121], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [110, 92], ActivityNetQA [122, 45], TGIFQA [52], and DramaQA [23].”
- Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
- ActBERT: Learning Global-Local Video-Text Representations (2020, background): “There are many existing video-and-language tasks to evaluate the model’s capacities in joint video-text representation learning, e.g., video question answering [38, 10, 19, 56], video captioning [48, 54], text-video retrieval [49, 43, 26], video grounding [52].”
- Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “, 2016) is based on text summaries, and TGIF-QA(Jang et al., 2017) uses predefined templates for question generation on short GIFs.”
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “These approaches have been extended to the video domain as well [20, 34, 42].”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Although our model is not dedicated for video tasks, our model achieve new SOTA on MSRVD, MSRVTT, and VATEX for captioning and on MSVD-QA and TGIF-Frame for QA.”
- From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “with subtitles; MovieQA [75], with videos from movies and questions obtained from higher-level plot summaries; PororoQA [43], with cartoon videos; and TGIFQA [39], with templated questions from the TGIF dataset [47].”
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, background): “Fortunately, large-scale video QA datasets have been recently emerged from the community using crowdsourcing on various sources of data (e.g. movies for MovieQA [10] and animated GIFs for TGIF-QA [11]).”
- TVQA: Localized, Compositional Video Question Answering (2018, methodology): “Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al.”
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “, 2017a) × X × × × × TGIF-QA (Jang et al., 2017) X × × × × × MovieQA (Tapaswi et al.”
- Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “For TGIF-QA, we compare with most recent SoTAs, including [6, 7, 10, 20], Model Appear.”
- Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (2019, methodology): “Many models have been proposed to tackle this problem [5, 10, 27, 30–32].”