Spatiotemporal-Textual Co-Attention Network for Video Question Answering

By Zhengjun Zha, Jiawei Liu, Tianhao Yang, Yongdong Zhang. 2019

48 citations recorded by the source corpus.

Visual Question Answering (VQA) is to provide a natural language answer for a pair of an image or video and a natural language question. Despite recent progress on VQA, existing works primarily focus on image question answering and are suboptimal for video question answering. This article presents a novel Spatiotemporal-Textual Co-Attention Network (STCA-Net) for video question answering. The STCA-Net jointly learns spatially and temporally visual attention on videos as well as textual attention on questions. It concentrates on the essential cues in both visual and textual spaces for answering question, leading to effective question-video representation. In particular, a question-guided attention network is designed to learn question-aware video representation with a spatial-temporal attention module. It concentrates the network on regions of interest within the frames of interest across the entire video. A video-guided attention network is proposed to learn video-aware question representation with a textual attention module, leading to fine-grained understanding of question. The learned video and question representations are used by an answer predictor to generate answers. Extensive experiments on two challenging datasets of video question answering, i.e., MSVD-QA and MSRVTT-QA, have shown the effectiveness of the proposed approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanwang Zhang
2. Meng Wang
3. Zhengjun Zha
4. Dan Guo
5. Yanlu Wei
6. Libo Zhang
7. Jay Zhangjie Wu
8. Haibo Wang

In Computer Vision and Pattern Recognition

1. Meng Wang
2. Hanwang Zhang
3. Zhengjun Zha
4. Xianglong Liu
5. Renshuai Tao
6. Dan Guo
7. Yanlu Wei
8. Libo Zhang

Citations in the corpus

• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “video domain (VideoQA), spatio-temporal video representations in terms of motion and appearance have been used in [22, 26, 31, 33, 34, 35, 40, 41, 42, 48, 84, 86, 94, 102].”
• Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “A vast majority of VideoQA approaches rely on relatively small, manually annotated VideoQA datasets [3, 10, 11, 16–18, 23, 26, 27, 33, 36, 37, 40–43, 49, 50, 53, 64, 67, 76, 77, 83, 88, 89, 93, 100, 108, 112, 115, 117, 124, 128].”
• Iterative Context-Aware Graph Inference for Visual Dialog (2020, background): “Recently, cross-modal semantic understanding between vision and language has attracted more and more interests, such as image captioning [35, 4, 36, 40, 20], referring expression [11, 41, 21], and visual question answering (VQA) [3, 18, 38, 39].”
• Occluded Prohibited Items Detection: An X-ray Security Inspection Benchmark and De-occlusion Attention Module (2020, background): “Attention can be interpreted as a means of biasing the allocation of available computational resources towards the most informative components of a signal, which has been widely studied in many tasks, like image retrieval [30, 40], visual question answering [20, 41].”