DeepStory: Video Story QA by Deep Embedded Memory Networks

By Kyung-Min Kim, Min-Oh Heo, Seongho Choi, Byoung-Tak Zhang. 2017

185 citations recorded by the source corpus.

Question-answering (QA) on video contents is a significant challenge for achieving human-level intelligence as it involves both vision and language in real-world settings. Here we demonstrate the possibility of an AI agent performing video story QA by learning from a large amount of cartoon videos. We develop a video-story learning model, i.e. Deep Embedded Memory Networks (DEMN), to reconstruct stories from a joint scene-dialogue video stream using a latent embedding space of observed data. The video stories are stored in a long-term memory component. For a given question, an LSTM-based attention model uses the long-term memory to recall the best question-story-answer triplet by focusing on specific words containing key information. We trained the DEMN on a novel QA dataset of children's cartoon video series, Pororo. The dataset contains 16,066 scene-dialogue pairs of 20.5-hour videos, 27,328 fine-grained sentences for scene description, and 8,913 story-related QA pairs. Our experimental results show that the DEMN outperforms other QA models. This is mainly due to 1) the reconstruction of video stories in a scene-dialogue combined form that utilize the latent embedding and 2) attention. DEMN also achieved state-of-the-art results on the MovieQA benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al.”
IQA: Visual Question Answering in Interactive Environments (2017, background): “These include datasets of natural images [2, 36, 41, 68], synthetic images [2, 4, 24, 27, 28], natural videos [23, 65], synthetic videos [32, 49] and multimodal contexts [29].”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “Inspired by recent works [30, 15, 18] which relied on Movie/Cartoon/TV shows for building multimodal datasets, we select TV shows as our data collection resource as they typically involve rich social interactions between actors, involving both activities and dialogues.”
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering (2019, methodology): “Several multi-modal videoQA datasets have been introduced including MovieQA [14], PororoQA [20], and TVQA [15].”
BERT Representations for Video Question Answering (2020, methodology): “We extensively evaluate our model on two well-known video-QA datasets: TVQA [21] and Pororo [18].”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Early studies on MVQA adopted an early-fusion framework [16, 23] that fuse video and subtitle into a joint embedding space at the early stage of the prediction pipeline which thereafter is the basis for subsequent reasoning followed by final prediction.”
TutorialVQA: Question Answering Dataset for Tutorial Videos (2019, background): “Accordingly, the answer format needs to also be improved towards more ﬂexible ways than multiple choice (Tapaswi et al., 2016; Jang et al., 2017) or ﬁll-in-the-blank questions (Maharaj et al., 2017; Kim et al., 2017).”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “DEMN [8] achieved best accuracy of 68% at the time of release.”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, background): “Several of these prominent benchmarks rely on dialogue and plot summaries instead of a video’s visual contents [33, 48, 27, 59], resulting in models with a stronger dependence on the dialogue than on the visual input and therefore reducing the benchmark’s effectiveness at measuring visual reasoning ”
Progressive Attention Memory Network for Movie Story Question Answering (2019, background): “Deep Embedded Memory Network (DEMN) [16] reconstructs stories from a joint stream of scene-dialogue using a latent embedding space and retrieves information which is relevant to the question.”
StoryGAN: A Sequential Conditional GAN for Story Visualization (2018, methodology): “We created two datasets from the existing CLEVR [19] and Pororo [21] datasets for our story visualization task, called CLEVR-SV and Pororo-SV, respectively.”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Existing Video QA datasets include MovieFIB [44], MovieQA [54], TGIF-QA [26], PororoQA [32], and TVQA [35, 36].”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, methodology): “To address video QA, several works have utilized early stage fusion method (Kim et al. 2017; Na et al. 2017) to merge two different modalities, while other recent works (Lei et al. 2018, 2019) have employed late stage fusion method, which extracts representation from language and vision…”
DramaQA: Character-Centered Video Story Understanding with Hierarchical QA (2020, methodology): “(Tapaswi et al. 2016; Kim et al. 2017; Mun et al. 2017; Jang et al. 2017; Lei et al. 2018) considered the video story QA task as an effective tool for multimodal story understanding and built video QA datsets.”