BERT Representations for Video Question Answering

By Zekun Yang, Noa García, Chenhui Chu, Mayu Otani, Yuta Nakashima, Haruo Takemura. 2020

118 citations recorded by the source corpus.

Visual question answering (VQA) aims at answering questions about the visual content of an image or a video. Currently, most work on VQA is focused on image-based question answering, and less attention has been paid into answering questions about videos. However, VQA in video presents some unique challenges that are worth studying: it not only requires to model a sequence of visual features over time, but often it also needs to reason about associated subtitles. In this work, we propose to use BERT, a sequential modelling technique based on Transformers, to encode the complex semantics from video clips. Our proposed model jointly captures the visual and language information of a video scene by encoding not only the subtitles but also a sequence of visual concepts with a pretrained language-based Transformer. In our experiments, we exhaustively study the performance of our model by taking different input arrangements, showing outstanding improvements when compared against previous work on two well-known video VQA datasets: TVQA and Pororo.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Pre-Trained Language Models for Interactive Decision-Making (2022, background): “Included in these successes are a number of tasks that integrate language and other modalities, including visual question answering and image captioning (Yang et al., 2020).”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “In contrast to some previous works using additional video features such as subtitles [13, 38, 39, 45, 46, 48, 77, 83, 90], our video representation is exclusively based on visual information, as we focus on the visual understanding of videos.”
Is Space-Time Attention All You Need for Video Understanding? (2021, background): “We also note that there is a wide literature based on the use of text Transformers combined with video CNNs to address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al., 2020) and dialog (Le et al., 2019).”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, methodology): “Several studies (Lei et al. 2019; Yang et al. 2020; Kim, Tang, and Bansal 2020; Kim et al. 2020) have taken advantages of these self-supervised pre-trained models and combined with their video QA models to learn representations of the text data such as questions, answers, subtitles, and extracted…”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[115] Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “Although video BERT approaches have been proposed to learn the correspondence between texts and video frames [27, 35, 49, 57], we are the first to integrate recurrence into BERT to learn partiallyobservable and temporal-dependent inputs.”
Sequence Parallelism: Long Sequence Training from System Perspective (2021, unknown): “Transformer-based language models [1, 2, 12] have achieved impressive performance on various natural language understanding and generation tasks ( e.g., Q&A [11, 23], relation extraction [21, 22, 27] and dialogue system [10]).”
Video Understanding as Machine Translation (2020, background): “ 38.8 VIDEOTRANSLATE (ours) yes: gen 11.6 43.9 14.7 52.8 Table 6: Visual Question Answering results on the TVQA dataset. Method Finetuning Subtitle as input Val Test* TVQA [23] yes no 45.03 - BERT QA [62] yes no 48.95 49.23 VIDEOTRANSLATE (ours) yes:gen no 58.72 58.39 VIDEOTRANSLATE (ours) yes:discr”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “ the experiment S. + Q. (w/ MUL. + w/ LSTM) to evaluate the effect of sequential modeling in textual stream. 16 Le et al. Exp. Model Validation w/o ts w/ ts State-of-the-art methods 1 TVQA S.+Q.+V.[2][46] 64.7 67.7 2 VideoQABERT[46] 63.1 72.1 Textual stream only 3 (B) Q. 41.6 41.6 4 (B) S. + Q. (w/o”
On Modality Bias in the TVQA Dataset (2020, unknown): “Yang et al. [45] present a detailed analysis of BERT on the TVQA dataset and propose a ‘V+S+Q+A’ model that is structurally similar to the simple TVQA baseline (i.e. separate visual and subtitle streams that additively combines their contributions at the voting stage).”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “However, [27] presents one of the first works which explores BERT in the domain of computer vision.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, methodology): “Such models have become ubiquitous; for example, visual representations from ImageNet [2] can be reused for tasks like cancer detection [3], and pre-trained language embeddings like BERT [4] have been used for everything from medical coding [5] to visual question answering [6].”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, methodology): “Most prior methods [25, 52, 29] rely on extra crossmodal fusion modules for combining input signals from different modalities.”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, unknown): “[59] 73.”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, methodology): “The applications include image retrieval [46], image captioning [1, 45], VQA [51, 25] and image generation [24, 19].”