Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA

By Hyounghun Kim, Zineng Tang, Mohit Bansal. 2020

31 citations recorded by the source corpus.

Videos convey rich information. Dynamic spatio-temporal relationships between people/objects, and diverse multimodal events are present in a video clip. Hence, it is important to develop automated models that can accurately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Specifically, we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra information (in explicit textual format to allow easier matching) for answering questions. Moreover, our model is also comprised of dual-level attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier. Finally, we also cast the frame selection problem as a multi-label classification task and introduce two loss functions, In-andOut Frame Score Margin (IOFSM) and Balanced Binary Cross-Entropy (BBCE), to better supervise the model with human importance annotations. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin (74.09% versus 70.52%). We also present several word, object, and frame level visualization studies.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “In contrast to some previous works using additional video features such as subtitles [13, 38, 39, 45, 46, 48, 77, 83, 90], our video representation is exclusively based on visual information, as we focus on the visual understanding of videos.”
Data augmentation techniques for the Video Question Answering task (2020, background): “Later on, attention mechanisms have been also used as a cross-modality fusion mechanism [11], and to learn QA-aware representations of both the visual and the textual data [20,16].”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, methodology): “More recently, beyond question answering on a single image, as video has become an important source of information, video QA has emerged as a key topic in the vision-language community (Lei et al. 2018, 2019; Kim et al. 2019b; Le et al. 2019; Kim, Tang, and Bansal 2020; Kim et al. 2020; Geng et al. ”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[40] Hyounghun Kim, Zineng Tang, and Mohit Bansal.”
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021, methodology): “Inspired by recent work (Kim and Bansal, 2019; Kim et al., 2020) that uses dense captions (Johnson et al., 2016; Yang et al., 2017) to improve image and video QA models, we propose to add dense cap-tions an as auxiliary text input that provide aligned visual cues to ease the difﬁculties of learning…”