Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
By Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, Heng Huang. 2019
319 citations recorded by the source corpus.
In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cordelia Schmid
- Mohit Bansal
- Trevor Darrell
- Steven C. H. Hoi
- Yoshua Bengio
- Luc Van Gool
- Chuang Gan
- Tat-seng Chua
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Cordelia Schmid
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
In Multimodal Learning
- Cordelia Schmid
- Chuang Gan
- Josef Sivic
- Josh Tenenbaum
- Ivan Laptev
- Antoine Miech
- Antonio Torralba
- Zhenfang Chen
In Question Answering & Reading Comprehension
Citations in the corpus
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “Using NExT-QA, we evaluate several state-of-the-art (SOTA) video QA techniques [9, 11, 18, 19, 23, 26].”
- Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “video domain (VideoQA), spatio-temporal video representations in terms of motion and appearance have been used in [22, 26, 31, 33, 34, 35, 40, 41, 42, 48, 84, 86, 94, 102].”
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, neutral): “Standard approaches [23, 74, 17, 80, 33, 12, 31, 32] leverage offline extracted video and text features from action recognition models [26, 66, 4, 73], image recognition models [10, 21], and language models [47, 50, 11, 42].”
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “We also include a recent model that incorporates heterogeneous memory with multimodal attention work ( Memory ) (Fan et al., 2019) that achieves superior performance on several datasets.”
- Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “Performance comparison on MSVD-QA and MSRVTT-QA dataset with state-of-the-art methods: Co-mem [7], HME [6], HRA [2], and AMU [39]. tions are of five types.”
- Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, methodology): “The results show that QueST outperforms the stateof-the-art methods (i.e. HME (Fan et al. 2019) and PSAC (Li et al. 2019)) by 2.0%, 3.2% and 4.0% of accuracy on action, transition and frame QA tasks.”
- Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “Multimodal learning is a well established methodology for tackling complex and challenging artificial intelligence tasks such as Visual Question Answering (Antol et al. 2015; Jang et al. 2017; Gao et al. 2018; Xu et al. 2017; Fan et al. 2019) and Text-to-Video Retrieval (Liu et al.”
- Look Before you Speak: Visually Contextualized Utterances (2020, neutral): “5 HMEMA [20] 33.”
- A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “Currently, the most common existing methods use attention [125,127,129] and memory [126] mechanisms to efficiently learn visual artifacts and the semantic correlations that allow questions to be answered accurately.”
- Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “All studies on Video QA treats short-form and longform Video QA as two separate problems in which proposed methods [8,1,14,16,17,2] are deviated to handle either one of the two problems.”
- AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, methodology): “Using AGQA, we evaluate modern visual reasoning systems (PSAC, HME, and HRCN [11, 33, 36]), and find that they barely perform better than models that purely exploit linguistic bias.”
- iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “, VQA, has been well explored [17, 2, 49, 1, 35, 36], the field of VideoQA is now picking up intense momentum [62, 22, 7, 30, 28].”
- Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, methodology): “Later, visual reasoning was extended to more complex dynamic videos (Lei et al., 2018; Fan et al., 2019; Li et al., 2020; 2019a; Huang et al., 2020).”
- EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, methodology): “Instead of simply applying temporal attention once to build the final answer abstraction, several work [21, 54, 60] adopted multi-step reasoning to better relate question words with encoded video features.”
- Reasoning with Heterogeneous Graph Alignment for Video Question Answering (2020, methodology): “For question, we follow previous work (Gao et al. 2018; Jang et al. 2019; Fan et al. 2019) and represent each word Figure 2: HGA network consists of four stages: representation, fusion, alignment, and reasoning.”
- Align and Prompt: Video-and-Language Pre-training with Entity Prompts (2021, methodology): “Most prior methods on text-video retrieval [14,32,43,53,55] and videoQA [12, 15, 25, 27] employ pre-trained visual back-bones and extract video features for each frame densely yet offline.”
- Neural Reasoning, Fast and Slow, for Video Question Answering (2019, methodology): “although we have not outperformed [7], [4], it is not directly comparable since they utilized motion in addition to appearance”