Video Question Answering via Hierarchical Spatio-Temporal Attention Networks

By Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang. 2017

117 citations recorded by the source corpus.

Open-ended video question answering is a challenging problem in visual information retrieval, which automatically generates the natural language answer from the referenced video content according to the question. However, the existing visual question answering works only focus on the static image, which may be ineffectively applied to video question answering due to the lack of modeling the temporal dynamics of video contents. In this paper, we consider the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We propose the hierarchical spatio-temporal attention network for learning the joint representation of the dynamic video contents according to the given question. We then develop the spatio-temporal attentional encoder-decoder learning method with multi-step reasoning process for open-ended video question answering. We construct a large-scale video question answering dataset. The extensive experiments show the effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Generation-based open-ended QA is of higher practical value and also receives widespread attention recently [53, 60, 61].”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “[29], that was notably used in [84, 93, 95, 96, 97] to generate VideoQA data from video descriptions.”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, background): “Developments along this direction include attribute-based attention [42], hierarchical attention [21, 45, 46], multi-head attention [14, 19], multi-step progressive attention memory [12] or combining self-attention with co-attention [20].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “Compared with existing works (Jang et al., 2017; Zhao et al., 2017) that use soft temporal attention, we use more interpretable hard attention, extracting local features (together with global features) for question answering.”
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “VQA approaches can be classiﬁed into the following methodological categories: the coarse joint-embedding models [39, 3, 11, 26], the ﬁne-grained joint-embedding models with attention [1, 18, 6, 9, 21, 38, 40] and the external knowledge based models [29, 30, 31].”
Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, methodology): “…Method what who how when where Average E-VQA [45] 0.189 0.387 0.835 0.705 0.292 0.264 E-SA [45] 0.220 0.416 0.796 0.731 0.332 0.293 E-MN [45] 0.234 0.418 0.837 0.708 0.276 0.304 DLAN [58] 0.254 0.428 0.810 0.721 0.312 0.320 GRAAM [45] concentrates on the crucial frames for answering question.”
Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks (2020, methodology): “Following the existing visual question answering approaches [3], [6], we generate five types of visual questions, that are related to object, number, location and action for the video content.”
Multi-agent Embodied Question Answering in Interactive Environments (2020, background): “VQA requires the agent to observe the given visual contents (i.e., images [1] or videos [13][23]) and reason out the answer combining the multi-modal inputs.”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, background): “Developments along this direction include attribute-based attention Ye et al. (2017), hierarchical attention Liang et al. (2018); Zhao et al. (2018, 2017), multi-head attention Kim et al. (2018); Li et al. (2019), multi-step progressive attention memory Kim et al. (2019) or combining self-attention…”
Attentive Moment Retrieval in Videos (2018, background): “And they also proposed a hierarchical spatiotemporal attention network [49] to learn the joint representation of the dynamic video contents according to the given question.”
Challenges and Prospects in Vision and Language Research (2019, background): “Video Question Answering has the additional requirement of understanding temporal dynamics (Zhu et al., 2017; Zhao et al., 2017b).”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “For the long-form VQA task, the proposed model is compared with state-of-the-art methods [STAN [6], MM+ [4] and ST-VQA [44]] on which the new model outperforms.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “Furthermore, open-ended video question answering is also addressed with methods such as spatio-temporal attentional encoder-decoder learning framework (Zhao et al., 2017).”
Video Question Answering via Gradually Refined Attention over Appearance and Motion (2017, unknown): “Recent papers [39, 49, 50] also employ the attention mechanism over videos.”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “Since we published a conference version of this work, there has been other related publications such as, Mari- oQA (Mun et al. 2017) and Hierarchical Spatio-Temporal Attention Networks (Zhao et al. 2017), that made use of spatio-temporal attention.”