Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering

By Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, Chuang Gan. 2019

299 citations recorded by the source corpus.

Most of the recent progresses on visual question answering are based on recurrent neural networks (RNNs) with attention. Despite the success, these models are often timeconsuming and having difficulties in modeling long range dependencies due to the sequential nature of RNNs. We propose a new architecture, Positional Self-Attention with Coattention (PSAC), which does not require RNNs for video question answering. Specifically, inspired by the success of self-attention in machine translation task, we propose a Positional Self-Attention to calculate the response at each position by attending to all positions within the same sequence, and then add representations of absolute positions. Therefore, PSAC can exploit the global dependencies of question and temporal information in the video, and make the process of question and video encoding executed in parallel. Furthermore, in addition to attending to the video features relevant to the given questions (i.e., video attention), we utilize the co-attention mechanism by simultaneously modeling “what words to listen to” (question attention). To the best of our knowledge, this is the first work of replacing RNNs with selfattention for the task of visual question answering. Experimental results of four tasks on the benchmark dataset show that our model significantly outperforms the state-of-the-art on three tasks and attains comparable result on the Count task. Our model requires less computation time and achieves better performance compared with the RNNs-based methods. Additional ablation study demonstrates the effect of each component of our proposed model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Using NExT-QA, we evaluate several state-of-the-art (SOTA) video QA techniques [9, 11, 18, 19, 23, 26].”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Current approaches to VideoQA rely on deep fully-supervised models trained on manually annotated datasets with question and answer pairs [23, 33, 36, 37, 42, 44, 50].”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “For TGIF-QA, we compare with most recent SoTAs, including [6, 7, 10, 20], Model Appear.”
Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, methodology): “(Li et al. 2019) employ attention mechanism, instead of LSTM or GRU, to generate question embedding and video feature embedding.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[64] Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan.”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “Developments along this direction include attribute-based attention [37], hierarchical attention [38,39,40], multi-head attention [41,42], multi-step progressive attention memory [17] or combining self-attention with co-attention [3].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Many video question answering datasets [25, 47, 40, 65, 79] and the methods [80, 49, 36, 73, 76, 20] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [65]) or 3D object movements without physical and language cues (CATER [25]).”
Video Graph Transformer for Video Question Answering (2022, background): “, cross-modal attention [20,33,22], motion-appearance memory [16,14,36], and graph neural networks [23,35,41], have been proposed to model informative videos contents for answering questions.”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “While some of them (Li et al. 2019; Gao et al. 2018a; Jang et al. 2017) principally focus on factual understanding in short videos, another research direction aims at understanding videos that contain story-lines and answering questions about them.”
Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, background): “[35], [36], [37], [38], [39], [40], [41], [42], [43] and some multi-modal reasoning benchmarks [11], [30], [44] have been developed.”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, methodology): “Self-attention mechanism has been widely used in many tasks, e.g., machine translation (Vaswani et al. 2017), image generation (Zhang et al. 2019) and video question answering (Li et al. 2019).”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, methodology): “PSAC uses positional self-attention and co-attention blocks to integrate visual and language features [36].”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, background): “PSAC [16] exploits the global dependencies of temporal information of video and question, executing video and question encoding processes in parallel.”
Reasoning with Heterogeneous Graph Alignment for Video Question Answering (2020, methodology): “The co-attention mechanism is viewed as another effective solution, and previous work has suggested a variety of task-speciﬁc structures (Nguyen and Okatani 2018; Gao et al. 2019; Yu et al. 2019; Li et al. 2019).”