Progressive Attention Memory Network for Movie Story Question Answering

By Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, Chang-Dong Yoo. 2019

80 citations recorded by the source corpus.

This paper proposes the progressive attention memory network (PAMN) for movie story question answering (QA). Movie story QA is challenging compared to VQA in two aspects: (1) pinpointing the temporal parts relevant to answer the question is difficult as the movies are typically longer than an hour, (2) it has both video and subtitle where different questions require different modality to infer the answer. To overcome these challenges, PAMN involves three main features: (1) progressive attention mechanism that utilizes cues from both question and answer to progressively prune out irrelevant temporal parts in memory, (2) dynamic modality fusion that adaptively determines the contribution of each modality for answering the current question, and (3) belief correction answering scheme that successively corrects the prediction score on each candidate answer. Experiments on publicly available benchmark datasets, MovieQA and TVQA, demonstrate that each feature contributes to our movie story QA architecture, PAMN, and improves performance to achieve the state-of-the-art result. Qualitative analysis by visualizing the inference mechanism of PAMN is also provided.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Hierarchical Conditional Relation Networks for Video Question Answering (2020, background): “Developments along this direction include attribute-based attention [42], hierarchical attention [21, 45, 46], multi-head attention [14, 19], multi-step progressive attention memory [12] or combining self-attention with co-attention [20].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, neutral): “46 PAMN (Kim et al., 2019b) 7 66.”
Neural Reasoning, Fast and Slow, for Video Question Answering (2019, unknown): “l of related information to the question. More sophisticated reasoning mechanisms are also developed with hierarchical attention[23], multi-head attention[16] or multistep progressive attention memory[15] to jointly reason on video/audio/text concurrent signals. Compared to the previous work, our fr”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, unknown): “[24] 66.”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, background): “More recently, beyond question answering on a single image, as video has become an important source of information, video QA has emerged as a key topic in the vision-language community (Lei et al. 2018, 2019; Kim et al. 2019b; Le et al. 2019; Kim, Tang, and Bansal 2020; Kim et al. 2020; Geng et al. ”
Multi-Modal Answer Validation for Knowledge-Based VQA (2021, background): “…has been used in various question answering settings, including textual QA (Zhang, Vu, and Moschitti 2021), product QA (Zhang et al. 2020), commonsense VQA (Wu, Chen, and Mooney 2020), movie QA (Kim et al. 2019), allowing systems to perform a more in-depth examination of each answer candidate.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “PAMN [20] includes 3 main features: progressive attention mechanism which uses clues from video and question to find the relevant answer, dynamic modality fusion and belief correction answering scheme that improves the prediction on every answer.”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “Video QA, instead of treating them as one big passage as in prior works [2,17], we dissect the subtitle passage S into a fixed number of M overlapping segments U = {U1, .”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, methodology): “Progressive Attention Memory Network (PAMN) [14] utilizes QA pairs to temporally attend video and subtitle memories, and merge using a soft attention mechanism.”
BERT Representations for Video Question Answering (2020, methodology): “…evidence in both the spatial and temporal domains; in [8] a video understanding task by fusing external knowledge and video-QA is introduced; in [15] a video question answering framework that requires to simultaneously retrieve the relevant moments and referenced visual concepts is proposed; in…”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “ES-MTL (Kim et al. 2019a) introduces additional temporal retrieval and modality alignment networks to predict the time when the question was generated and to ﬁnd associations of video and subtitles.”
Reasoning with Heterogeneous Graph Alignment for Video Question Answering (2020, background): “More recently, part of the contributions applied the dynamic memory network to enhance the intelligence through better representation and fusion strategies (Wang et al. 2018; Gao et al. 2018; Fan et al. 2019; Kim et al. 2019).”