Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering

By Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, Chang-Dong Yoo. 2019

28 citations recorded by the source corpus.

This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answering is expensive and the existing benchmarks are relatively small to provide sufficient supervision. To overcome this challenge, this paper proposes a multi-task learning method which is composed of three main components: (1) multi-modal video question answering network that answers the question based on the both video and subtitle feature, (2) temporal retrieval network that predicts the time in the video clip where the question was generated from and (3) modality alignment network that solves metric learning problem to find correct association of video and subtitle modalities. By simultaneously solving related auxiliary tasks with hierarchically shared intermediate layers, the extra synergistic supervisions are provided. Motivated by curriculum learning, multi-task ratio scheduling is proposed to learn easier task earlier to set inductive bias at the beginning of the training. The experiments on publicly available dataset TVQA shows state-of-the-art results, and ablation studies are conducted to prove the statistical validity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, neutral): “77 multi-task (Kim et al., 2019a) 3 66.”
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA (2020, unknown): “38 4 Multi-task (Kim et al., 2019a) 67.”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, background): “…beyond question answering on a single image, as video has become an important source of information, video QA has emerged as a key topic in the vision-language community (Lei et al. 2018, 2019; Kim et al. 2019b; Le et al. 2019; Kim, Tang, and Bansal 2020; Kim et al. 2020; Geng et al. 2020).”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, methodology): “Multi-task Learning (MTL) [13] further extends the”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, background): “video captioning [26, 41, 5], video object segmentation [7, 13] and video question answering [19, 14, 35].”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “ES-MTL (Kim et al. 2019a) introduces additional temporal retrieval and modality alignment networks to predict the time when the question was generated and to ﬁnd associations of video and subtitles.”