Leveraging Video Descriptions to Learn Video Question Answering

By Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, Min Sun. 2016

195 citations recorded by the source corpus.

We propose a scalable approach to learn video-based question answering (QA): to answer a free-form natural language question about the contents of a video. Our approach automatically harvests a large number of videos and descriptions freely available online. Then, a large number of candidate QA pairs are automatically generated from descriptions rather than manually annotated. Next, we use these candidate QA pairs to train a number of video-based QA methods extended from MN (Sukhbaatar et al. 2015), VQA (Antol et al. 2015), SA (Yao et al. 2015), and SS (Venugopalan et al. 2015). In order to handle non-perfect candidate QA pairs, we propose a self-paced learning procedure to iteratively identify them and mitigate their effects in training. Finally, we evaluate performance on manually generated video-based QA pairs. The results show that our self-paced learning procedure is effective, and the extended SS model outperforms various baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Established works [18, 42, 59, 51] apply 2D convolutional neural networks (CNNs) (e.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “[29], that was notably used in [84, 93, 95, 96, 97] to generate VideoQA data from video descriptions.”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “To increase the memorizing ability, ex-ternal memory can be added to these networks [7, 44].”
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (2019, background): “Several recent work [5, 15, 32] tried to augment VideoQA with differently embodied memory networks [23, 25, 26].”
Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks (2018, methodology): “As a natural extension of image-based question answering, the video-based question answering has been proposed as a more challenging task [Zeng et al. , 2016].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Other works (Zeng et al., 2017) have modified it to support answering free-form natural language questions.”
BERT Representations for Video Question Answering (2020, background): “…retrieve the relevant moments and referenced visual concepts is proposed; in [49] video-QA is studied in the temporal domain to do inference and prediction; and in [47] a scalable approach to automatically harvest videos and descriptions online and generate candidate QA pairs is proposed.”
Video Question Answering via Gradually Refined Attention over Appearance and Motion (2017, methodology): “To overcome the weakness of the models proposed in [43], we propose an end-to-end model for VideoQA.”
A Better Way to Attend: Attention With Trees for Video Question Answering (2018, methodology): “We compare our model with the E-SA [41] model and show a significant improvement.”
TutorialVQA: Question Answering Dataset for Tutorial Videos (2019, background): “Consequentially, videos have been increasingly used as main data sources in many question answering problems (Yang et al., 2003; Tapaswi et al., 2016; Jang et al., 2017; Maharaj et al., 2017; Kim et al., 2017; Jang et al., 2017; Zeng et al., 2017).”
Neural Reasoning, Fast and Slow, for Video Question Answering (2019, methodology): “Neural reasoning for Video QA: The work in [19], [32] both utilized memory network as a platform to retrieve the information in the video features related to the question embedding.”
Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, background): “Lots of methods have attempted to address the multimodal learning problems through dealing with visualsemantic learning tasks, such as visual question answering [32, 56, 54] and image captioning [10, 49, 12].”
Multi-interaction Network with Object Relation for Video Question Answering (2019, background): “Although, there are several other video QA datasets, such as MovieQA [30], MSRVTT-QA [34] and Video-QA [41], however, all of them use question types to divide the dataset into sub-tasks, which is influenced by the ideas of image QA task and cannot reflect the dynamic characteristic of video informat”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, methodology): “standing visual information conditioned on language. While QA based on a single image, i.e., VQA, has been well explored [17, 2, 49, 1, 35, 36], the ﬁeld of VideoQA is now picking up intense momentum [62, 22, 7, 30, 28]. While VQA relies on spatial information, VideoQA requires an understanding of b”
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering (2020, unknown): “entation is traditionally done by variations of recurrent networks (RNNs) among which many were used for Video QA such as recurrent encoderdecoder [18,19], bidirectional LSTM [16] and two-staged LSTM [20]. To increase the memorizing ability, external memory can be added to these networks [1,20]. Thi”
Multichannel Attention Refinement for Video Question Answering (2020, methodology): “Reference [61] simply extended several existing models from other”
Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks (2020, methodology): “Given a natural-language question for the video, the task of video-based question answering is to provide the accurate answer for the given question [44].”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, background): “Other video-only benchmarks suffer from the biases and simplicity associated with human generated questions [58, 48, 18, 33] or descriptions [53, 61].”