Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering

By Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, Yue Gao. 2020

131 citations recorded by the source corpus.

Understanding questions and finding clues for answers are the key for video question answering. Compared with image question answering, video question answering (Video QA) requires to find the clues accurately on both spatial and temporal dimension simultaneously, and thus is more challenging. However, the relationship between spatio-temporal information and question still has not been well utilized in most existing methods for Video QA. To tackle this problem, we propose a Question-Guided Spatio-Temporal Contextual Attention Network (QueST) method. In QueST, we divide the semantic features generated from question into two separate parts: the spatial part and the temporal part, respectively guiding the process of constructing the contextual attention on spatial and temporal dimension. Under the guidance of the corresponding contextual attention, visual features can be better exploited on both spatial and temporal dimensions. To evaluate the effectiveness of the proposed method, experiments are conducted on TGIF-QA dataset, MSRVTT-QA dataset and MSVD-QA dataset. Experimental results and comparisons with the state-of-the-art methods have shown that our method can achieve superior performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Mohit Bansal
3. Lijuan Wang
4. Jianfeng Gao
5. Zicheng Liu
6. Linjie Li
7. Cordelia Schmid
8. Jingjing Liu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Zhe Gan
4. Dacheng Tao
5. Trevor Darrell
6. Andrew Zisserman
7. Jitendra Malik
8. Xiaogang Wang

In Question Answering & Reading Comprehension

1. Mohit Bansal
2. Licheng Yu
3. Jie Lei
4. Tamara L. Berg
5. Zhe Gan
6. Devshree Patel
7. Ratnam Parikh
8. Yesha Shastri

Citations in the corpus

• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “…E-SA [87] 29.3 27.6 ST-TP [35] 30.9 31.3 AMU [87] 32.5 32.0 Co-mem [27] 32.0 31.7 HME [23] 33.0 33.7 LAGCN [33] — 34.3 HGA [37] 35.5 34.7 QueST [36] 34.6 36.1 HCRN [42] 35.6 36.1 ClipBERT [44] COCO [15]+ Visual Genome [41] 37.4 — sistent and significant improvements for all datasets when…”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[49] Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao.”
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “However, existing works [41, 26, 29, 20, 50, 44] in this field are often evaluated on distinct datasets under different experimental settings, making fair comparison difficult between methods.”
• NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, background): “[19] proposed a question-guided spatial-temporal contextual attention network, and Qian et al.”
• Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “In QueST [28], semantic features are divided into spatial and temporal parts, which are then used”
• Video Graph Transformer for Video Question Answering (2022, background): “, cross-modal attention [20,33,22], motion-appearance memory [16,14,36], and graph neural networks [23,35,41], have been proposed to model informative videos contents for answering questions.”
• Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[36] Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao.”