Multi-interaction Network with Object Relation for Video Question Answering

By Weike Jin, Zhou Zhao, Mao Gu, Jun Yu, Jun Xiao, Yueting Zhuang. 2019

60 citations recorded by the source corpus.

Video question answering is an important task for testing machine's ability of video understanding. The existing methods normally focus on the combination of recurrent and convolutional neural networks to capture spatial and temporal information of the video. Recently, some work has also shown that using attention mechanism can achieve better performance. In this paper, we propose a new model called Multi-interaction network for video question answering. There are two types of interactions in our model. The first type is the multi-modal interaction between the visual and textual information. The second type is the multi-level interaction inside the multi-modal interaction. Specifically, instead of using original self-attention, we propose a new attention mechanism called multi-interaction, which can capture both element-wise and segment-wise sequence interactions, simultaneously. And in addition to the normal frame-level interaction, we also take the object relations into consideration, in order to obtain more fine-grained information, such as motions and other potential relations among these objects. We evaluate our method on TGIF-QA and other two video QA datasets. The qualitative and quantitative experimental results show the effectiveness of our model, which achieves the new state-of-the-art performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tat-seng Chua
2. Junbin Xiao
3. Truyen Tran
4. Svetha Venkatesh
5. Angela Yao
6. Vuong Le
7. Devshree Patel
8. Ratnam Parikh

In Computer Vision and Pattern Recognition

1. Tat-seng Chua
2. Junbin Xiao
3. Angela Yao
4. Xindi Shang
5. Svetha Venkatesh
6. Vuong Le
7. Devshree Patel
8. Ratnam Parikh

In Multimodal Learning

1. Svetha Venkatesh
2. Angela Yao
3. Tat-seng Chua
4. Vuong Le
5. Xindi Shang
6. Thao Minh Le
7. Truyen Tran
8. Junbin Xiao

In Question Answering & Reading Comprehension

1. Tat-seng Chua
2. Junbin Xiao
3. Svetha Venkatesh
4. Angela Yao
5. Vuong Le
6. Devshree Patel
7. Ratnam Parikh
8. Yesha Shastri

Citations in the corpus

• NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “, RNNs with LSTM [15], GRU [5] and their variants) and attention [26, 20].”
• Hierarchical Conditional Relation Networks for Video Question Answering (2020, background): “Relations of predetected objects were also considered in a separate processing stream [11] and combined with other modalities in late-fusion [29].”
• Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, background): “, 2019 [19] proposes a novel multi-interaction network that uses two different types of interactions, i.”
• Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “predetected objects were also considered in a separate processing stream [29] and combined with other modalities in late-fusion [30].”