Modality Shifting Attention Network for Multi-Modal Video Question Answering
By Junyeong Kim, Minuk Ma, Trung X. Pham, Kyungsu Kim, Chang-Dong Yoo. 2020
80 citations recorded by the source corpus.
This paper considers a network referred to as Modality Shifting Attention Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN decomposes the task into two sub-tasks: (1) localization of temporal moment relevant to the question, and (2) accurate prediction of the answer based on the localized moment. The modality required for temporal localization may be different from that for answer prediction, and this ability to shift modality is essential for performing the task. To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most appropriate temporal moment from each of the modalities, and also on (2) the heterogeneous reasoning network (HRN) that predicts the answer using an attention mechanism on both modalities. MSAN is able to place importance weight on the two modalities for each sub-task using a component referred to as Modality Importance Modulation (MIM). Experimental results show that MSAN outperforms previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark dataset. Extensive ablation studies and qualitative analysis are conducted to validate various components of the network.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cordelia Schmid
- Ivan Laptev
- Josef Sivic
- Lorenzo Torresani
- Andrew Zisserman
- Shih-Fu Chang
- Luc Van Gool
- Li Fei-Fei
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Cordelia Schmid
- Dacheng Tao
- Andrew Zisserman
- Yi Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
In Multimodal Learning
- Cordelia Schmid
- Josef Sivic
- Ivan Laptev
- Antoine Miech
- Lorenzo Torresani
- Antoine Yang
- Shih-Fu Chang
- Xudong Lin
In Question Answering & Reading Comprehension
- Jun Liu
- Ivan Laptev
- Devi Parikh
- Lorenzo Torresani
- Josef Sivic
- Antoine Miech
- Seohyeong Jeong
- Antoine Yang
Citations in the corpus
- Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “In contrast to some previous works using additional video features such as subtitles [35, 36, 40, 41, 43, 69, 78], our video representation is exclusively based on visual information.”
- iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, unknown): “[25] 71.”
- Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, methodology): “Several studies (Lei et al. 2019; Yang et al. 2020; Kim, Tang, and Bansal 2020; Kim et al. 2020) have taken advantages of these self-supervised pre-trained models and combined with their video QA models to learn representations of the text data such as questions, answers, subtitles, and extracted…”
- VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “Several approaches have been introduced for these tasks [34, 3, 23, 42, 16, 41, 25, 52, 20, 27, 30, 55, 17, 31].”
- Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, methodology): “Moreover, since iterative cross-modal interaction is also important for handling multi-modal reasoning [1], [2], [34], in our network,”
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[41] Junyeong Kim, Minuk Ma, Trung Pham, Kyungsu Kim, and Chang D Yoo.”