DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

By Seongho Choi, Kyoung-Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Seungchan Lee, Byoung-Tak Zhang. 2020

69 citations recorded by the source corpus.

Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story. Moreover, researches on how to evaluate the degree of video understanding based on human cognitive process have not progressed as yet. In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focuses on two perspectives: 1) Hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) Character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama "Another Miss Oh" and it contains 17,983 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors and emotions of main characters, and coreference resolved scripts. Additionally, we suggest Multi-level Context Matching model which hierarchically understands character-centered representations of video to answer questions. We release our dataset and model publicly for research purposes, and we expect our work to provide a new perspective on video story understanding research.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “We report results on 12 video reasoning tasks: TVQA [64], TVQA(+) [65], VLEP [66], MSRVTT-QA [117], MSRVTT-Multichoice [121], LSMDC-Multichoice, LSMDC ﬁll-in-the-blank QA [110, 92], ActivityNetQA [122, 45], TGIFQA [52], and DramaQA [23].”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[11] Seongho Choi, Kyoung-Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Minsu Lee, and Byoung-Tak Zhang.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “[16] Seongho Choi, Kyoung-Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Seungchan Lee, Minsu Lee, and Byoung-Tak Zhang.”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, methodology): “• We show that our framework achieves state-of-the-art performance on TVQA, TVQA+, and DramaQA.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “DramaQA [43] 23,928 16,191 Automatic and Manual TV Drama Multiple-choice”