Character Matters: Video Story Understanding with Character-Aware Relations
By Shijie Geng, Ji Zhang, Zuohui Fu, Peng Gao, Hang Zhang, Gerard de Melo. 2020
11 citations recorded by the source corpus.
Different from short videos and GIFs, video stories contain clear plots and lists of principal characters. Without identifying the connection between appearing people and character names, a model is not able to obtain a genuine understanding of the plots. Video Story Question Answering (VSQA) offers an effective way to benchmark higher-level comprehension abilities of a model. However, current VSQA methods merely extract generic visual features from a scene. With such an approach, they remain prone to learning just superficial correlations. In order to attain a genuine understanding of who did what to whom, we propose a novel model that continuously refines character-aware relations. This model specifically considers the characters in a video story, as well as the relations connecting different characters and objects. Based on these signals, our framework enables weakly-supervised face naming through multi-instance co-occurrence matching and supports high-level reasoning utilizing Transformer structures. We train and test our model on the six diverse TV shows in the TVQA dataset, which is by far the largest and only publicly available dataset for VSQA. We validate our proposed approach over TVQA dataset through extensive ablation study.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Natural Language Processing
In Multimodal Learning
Citations in the corpus
- RomeBERT: Robust Training of Multi-Exit BERT (2021, background): “Transformer has achieved success on both visual and textual understanding (Devlin et al., 2019; Wang et al., 2018; Tan and Bansal, 2019; Geng et al., 2020b,a).”
- Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, background): “More recently, beyond question answering on a single image, as video has become an important source of information, video QA has emerged as a key topic in the vision-language community (Lei et al. 2018, 2019; Kim et al. 2019b; Le et al. 2019; Kim, Tang, and Bansal 2020; Kim et al. 2020; Geng et al. ”