Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks

By Zhou Zhao, Shuwen Xiao, Zehan Song, Chujie Lu, Jun Xiao, Yueting Zhuang. 2020

35 citations recorded by the source corpus.

As a challenging task in visual information retrieval, open-ended long-form video question answering automatically generates the natural language answer from the referenced video content according to the given question. However, the existing video question answering works mainly focus on the short-form video, which may be ineffectively applied for long-form video question answering directly, due to the insufficiency of modeling the semantic representation of long-form video content. In this paper, we study the problem of open-ended long-form video question answering from the viewpoint of hierarchical multi-modal conditional adversarial network learning. We propose the hierarchical attentional encoder network to learn the joint representation of long-form video content and given question with adaptive video segmentation. We then devise the reinforced decoder network to generate the natural language answer for open-ended video question answering with multi-modal conditional adversarial network learning. We construct three large-scale open-ended video question answering datasets. The extensive experiments validate the effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Devshree Patel
2. Ratnam Parikh
3. Yesha Shastri
4. Bourahla Omar
5. Abdul Jabbar
6. Xi Li

In Computer Vision and Pattern Recognition

1. Devshree Patel
2. Ratnam Parikh
3. Yesha Shastri
4. Bourahla Omar
5. Abdul Jabbar
6. Xi Li

Citations in the corpus

• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “To address this issue, several works have explored leveraging manually annotated video descriptions [35, 82, 87, 96, 98, 99, 100] for automatic generation of VideoQA datasets, using rule-based [30, 66] approaches.”
• Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “The authors of [24] implement a hierarchical attentional encoder network using adaptive video segmentation which jointly extracts the video representations with long-term dependencies based on the questions.”
• A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “GANs models have also been utilized in many other video applications such as Pose-GAN (PGAN) [172] and Dual-Motion GAN (DMGAN) [173] for future frame (video) prediction; Recycle-GAN (RGAN) [174] for video retargeting, and Video Question Answering via Multi-Modal CGAN (VQAMM-CGAN) [175] for video que”