A Better Way to Attend: Attention With Trees for Video Question Answering

By Hongyang Xue, Wenqing Chu, Zhou Zhao, Deng Cai. 2018

34 citations recorded by the source corpus.

We propose a new attention model for video question answering. The main idea of the attention models is to locate on the most informative parts of the visual data. The attention mechanisms are quite popular these days. However, most existing visual attention mechanisms regard the question as a whole. They ignore the word-level semantics where each word can have different attentions and some words need no attention. Neither do they consider the semantic structure of the sentences. Although the extended soft attention model for video question answering leverages the word-level attention, it performs poorly on long question sentences. In this paper, we propose the heterogeneous tree-structured memory network (HTreeMN) for video question answering. Our proposed approach is based upon the syntax parse trees of the question sentences. The HTreeMN treats the words differently where the visual words are processed with an attention module and the verbal ones not. It also utilizes the semantic structure of the sentences by combining the neighbors based on the recursive structure of the parse trees. The understandings of the words and the videos are propagated and merged from leaves to the root. Furthermore, we build a hierarchical attention mechanism to distill the attended features. We evaluate our approach on two data sets. The experimental results show the superiority of our HTreeMN model over the other attention models, especially on complex questions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Cordelia Schmid
2. Ivan Laptev
3. Josef Sivic
4. Antoine Miech
5. Andrew Zisserman
6. Jun Xiao
7. Zhou Zhao
8. Yueting Zhuang

In Computer Vision and Pattern Recognition

1. Cordelia Schmid
2. Luc Van Gool
3. Andrew Zisserman
4. Ivan Laptev
5. Dacheng Tao
6. Yi Yang
7. Jitendra Malik
8. Josef Sivic

In Question Answering & Reading Comprehension

1. Cordelia Schmid
2. Yueting Zhuang
3. Jun Xiao
4. Zhou Zhao
5. Antoine Miech
6. Josef Sivic
7. Ivan Laptev
8. Antoine Yang

Citations in the corpus

• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “More specifically to the video domain (VideoQA), spatio-temporal video representations in terms of motion and appearance have been used in [23, 27, 33, 35, 36, 37, 42, 43, 44, 50, 87, 89, 97, 105].”
• Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[112] Hongyang Xue, Wenqing Chu, Zhou Zhao, and Deng Cai.”
• Multichannel Attention Refinement for Video Question Answering (2020, background): “In Reference [49], a heterogeneous tree-structured memory network is proposed to perform attention based on the syntax parse trees of the question sentences.”