Motion-Appearance Co-memory Networks for Video Question Answering

By Jiyang Gao, Runzhou Ge, Kan Chen, Ramkant Nevatia. 2018

267 citations recorded by the source corpus.

Video Question Answering (QA) is an important task in understanding video temporal structure. We observe that there are three unique attributes of video QA compared with image QA: (1) it deals with long sequences of images containing richer information not only in quantity but also in variety; (2) motion and appearance information are usually correlated with each other and able to provide useful attention cues to the other; (3) different questions require different number of frames to infer the answer. Based on these observations, we propose a motion-appearance co-memory network for video QA. Our networks are built on concepts from Dynamic Memory Network (DMN) and introduces new mechanisms for video QA. Specifically, there are three salient aspects: (1) a co-memory attention mechanism that utilizes cues from both motion and appearance to generate attention; (2) a temporal conv-deconv network to generate multi-level contextual facts; (3) a dynamic fact ensemble method to construct temporal representation dynamically for different questions. We evaluate our method on TGIF-QA dataset, and the results outperform state-of-the-art significantly on all four tasks of TGIF-QA.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “video domain (VideoQA), spatio-temporal video representations in terms of motion and appearance have been used in [22, 26, 31, 33, 34, 35, 40, 41, 42, 48, 84, 86, 94, 102].”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, neutral): “Standard approaches [23, 74, 17, 80, 33, 12, 31, 32] leverage offline extracted video and text features from action recognition models [26, 66, 4, 73], image recognition models [10, 21], and language models [47, 50, 11, 42].”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “Performance comparison on MSVD-QA and MSRVTT-QA dataset with state-of-the-art methods: Co-mem [7], HME [6], HRA [2], and AMU [39]. tions are of ﬁve types.”
Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, methodology): “Most recent methods in Visual QA (Anderson et al. 2018; Jang et al. 2017; Gao et al. 2018; Li et al. 2019; Fan et al. 2019) adopt CNN methods, such as Faster R-CNN (Ren et al. 2015), ResNet (He et al. 2016), C3D (Tran et al. 2015), flow CNN, as the visual feature extractor.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “Multimodal learning is a well established methodology for tackling complex and challenging artificial intelligence tasks such as Visual Question Answering (Antol et al. 2015; Jang et al. 2017; Gao et al. 2018; Xu et al. 2017; Fan et al. 2019) and Text-to-Video Retrieval (Liu et al.”
Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “o state-of-the-art on MSVD-QA and MSRVTT-QA. We report top 1 accuracy [%]. We show results of our model with and without pretraining on How2FUP. Methods MSVD-QA MSRVTT-QA ST-VQA [35] 31.3 30.9 Co-Mem [27] 31.7 32.0 AMU [85] 32.0 32.5 HMEMA [20] 33.7 33.0 HRA [13] 34.4 35.1 SSML [3] 35.1 35.1 HCRN [4”
Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, methodology): “[9] designed a co-memory attention mechanism to jointly model motion and appearance information and used the temporal conv-deconv networks to build multi-level contextual facts for VideoQA.”
Align and Prompt: Video-and-Language Pre-training with Entity Prompts (2021, methodology): “Most prior methods on text-video retrieval [14,32,43,53,55] and videoQA [12, 15, 25, 27] employ pre-trained visual back-bones and extract video features for each frame densely yet ofﬂine.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “, 2019) In contrast to previous works like [10], [16] proposes Positional Self Attention with Co-attention (PSAC) architecture without recurrent neural networks for addressing the problem of VQA.”
Neural Reasoning, Fast and Slow, for Video Question Answering (2019, methodology): “d hugely in recent years [7], deep video understanding remains elusive. Compared to static images, video poses new challenges, primarily due to the inherent dynamic nature of visual content over time [6, 34]. At the lowest level, we have correlated motion and appearance [6]. At a higher level, we ha”
Multichannel Attention Refinement for Video Question Answering (2020, methodology): “Most of the existing works either encode the question as a single semantic vector [9, 61] or fuse the extracted appearance and motion features via average pooling or concating [17, 55].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, background): “Several problems are at the intersection between language and video: captioning [24, 38, 55, 59], retrieval [9, 16, 21, 31, 33, 52, 54] and visual question answering [15, 56, 57, 61].”
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering (2019, methodology): “proposed a co-memory network to jointly model and interact with the motion and appearance information (Gao et al. 2018).”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “While some of them (Li et al. 2019; Gao et al. 2018a; Jang et al. 2017) principally focus on factual understanding in short videos, another research direction aims at understanding videos that contain story-lines and answering questions about them.”
EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, methodology): “[27] applied late fusion on different types of video features and integrated them with memory modules.”
Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks (2020, background): “appearance and motion channels [7], [32], [47] are proposed to learn the question-aware semantic representation.”